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This  publication  is  primarily  a  working  paper.  It  is  published  solely  to  document  work  performed. 


SUMMARY 


In  the  current  environment  of  high  deficits  and  pressure  to  reduce  spending,  it  is  critical 
that  the  Air  Force  use  its  resources  as  efficiently  and  effectively  as  possible.  Foremost  among 
these  resources  are  Air  Force  personnel,  and  the  skills,  talents,  experience,  and  capacities  they 
possess,  in  order  to  minimize  the  “costs"  of  competing  activities,  the  Air  Force  must  assess 
individuals'  abilities  to  complete  the  requirements  of  jobs  within  occupational  specialties  and 
establish  criteria  for  aliowing  access  to  certain  jobs  or  specialties. 

The  Time  to  Proficiency  (TTP)  Project  had  two  major  objectives: 

1 .  to  determine  the  feasibility  and  validity  of  using  supervisor  estimates  of  relative  performance 
and  supervision  time  as  measures  of  performance  and  to  estimate  the  extent  to  which  mental 
aptitudes  affect  job  performance 

2.  to  develop  and  apply  a  prototype  analytical  model  to  evaluate  how  changing  aptitude 
requirements  based  on  job  performance  information  would  affect  occupational  capability; 
manpower,  personnel,  and  training  (MPT)  policies;  MPT  programs;  and  MPT  costs. 

in  order  to  achieve  these  objectives,  one  specialty.  Avionic  Communications  Specialist  (AFS 
328X0),  was  selected  and  data  were  collected  from  supervisors  concerning  the  relative  proficiency 
of  their  subordinates.  Data  were  also  collected  on  the  airmen’s  experience  and  mental  aptitude 
(as  measured  by  the  Armed  Services  Vocational  Aptitude  Battery  •  ASVAB).  These  datd  were 
then  anaiyzed  using  multivariate  regression  techniques  to  estimate  the  effects  of  aptitude  on 
productivity,  controlling  for  experience.  Attrition  information  obtained  from  the  Defense  Manpower 
Data  Center  (DMDC)  was  used  to  model  the  likelihood  of  remaining  in  service  as  a  function 
of  aptitude  and  time.  First-term  costs  were  modeled  using  average  recruiting  and  training  costs 
and  military  pay.  These  models  were  integrated  to  arrive  at  an  objective  model  which  minimizes 
cost  per  productive  unit  with  respect  to  aptitude. 

The  objective  model,  as  well  as  its  components,  yielded  significant  findings  for  AFS  328X0: 

1.  Higher  quality  people  achieve  a  specified  level  of  productivity  more  rapidly. 

2.  The  quaiity/experience  trade-off  is  approximately  16  ASVAB  electronics  score  points  per 
10  months  of  experience. 

3.  The  attrition  model  confirms  that  the  loss  rate  during  training  is  higher  for  the  lower 
aptitude  groups. 

4.  Using  a  simplified  cost  model  which  assumes  no  differences  in  recruiting,  training  or 
sustainment  costs  over  the  first  term  as  a  function  of  quality,  high  quality  people  are  more 
cost  effective  over  the  first  term. 

The  present  findings  suggest  that  TTP  is  a  potentially  viable  performance  measurement 
technique  which  is  relatively  easy  to  administer  and  provides  sufficient  information  for  the 
modeling  of  productivity.  The  project  also  demonstrated  how  an  objective  model  can  be 
formulated  and  solved  to  arrive  at  an  optimum  level  of  aptitude:  however,  further  refinement  is 
needed. 
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The  implication  of  these  results,  for  the  one  AFS  studied,  is  that  it  would  be  cost  effective 
for  the  Air  Force  to  recruit  people  at  the  highest  possible  quality  level.  To  make  Air  Force- wide 
conclusions,  larger  samples  and  more  AFSs  should  be  examined. 
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PREFACE 


The  Time  to  Proficiency  (TTP)  study  is  part  of  a  large-scale  research  and 
development  (R&D)  effort  to  develop  a  measurement  technology  for  systematically 
obtaining  job  performance  data.  This  R&D  effort  was  undertaken  by  the  Air  Force 
Human  Resources  Laboratory  (AFHRL)  to  satisfy  three  primary  requirements: 

1 .  Assist  the  manpower,  personnel  and  training  (MPT)  community  in  the  evaluation 
of  training  and  selection  programs, 

2.  Develop  criteria  for  other  MPT  R&D  projects,  and 

3.  Test  the  feasibility  of  validating  the  Armed  Services  Vocational  Aptitude  Battery 
(ASVAB)  against  job  performance  measures. 

TTP  utilizes  time-based  supervisor  rating  forms  to  build  an  aptitude/experience 
performance  model. 

We  wish  to  express  our  appreciation  to  those  who  contributed  to  this  project.  In 
particular  we  wish  to  thank  Dr.  Craig  Moore  of  the  Rand  Corporation  for  his  conceptual 
work  prior  to  this  effort.  We  also  wish  to  thank  Captain  Mark  Reid  of  the  Defense 
Manpower  Data  Center  for  his  assistance  in  the  acquisition  of  demographic  information 
for  the  measured  incumbents  and  attrition  data  for  the  1982  cohort  group. 
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TIME  TO  JOB  PROFICIENCY:  A  PRELIMINARY 
INVESTIGATION  OF  THE  EFFECTS  OF  APTITUDE  AND 
EXPERIENCE  ON  PRODUCTIVE  CAPACITY 


I.  INTRODUCTION 

Evaluating  the  influence  of  mental  aptitude  on  job  performance  in  individual  specialties  is 
an  important  issue  facing  the  Air  Force.  This  evaluation  wouid  faciiitate  identifying  aptitude 
requirements  for  each  specialty.  These  requirements  should  balance  the  work  force's  operational 
capability  and  the  practicality  of  managing  the  work  force’s  manpower,  personnel,  and  training 
costs  in  an  efficient,  albeit  optimal,  manner.  This  objective  is  constrained  by  the  distribution 
of  mentai  aptitudes  among  the  pooi  from  which  new  members  are  recruited. 

The  Time  to  Proficiency  (TTP)  project  is  based  on  a  unique  approach  to  the  baiancing  of 
aptitude  requirements  and  cost  and  productivity  requirements.  The  approach  features  the  notion 
of  "time  to  proficiency,"  the  length  of  time  it  takes  to  bring  people  with  different  attributes 
(especialiy  mental  aptitude)  to  targeted  ieveis  of  task  performance.  The  hypothesis  is  simple; 
The  higher  the  ieveis  of  aptitude  required  for  entry  into  a  speciaity,  the  faster  the  recruits  will 
achieve  job  proficiency  and  the  greater  the  capacity  of  the  work  force  emplaced. 

However,  if  aptitude  requirements  are  raised,  recruiting  and  retention  costs  may  also  increase. 
The  Air  Force’s  objective  is  to  find  those  levels  of  aptitude  requirements  which  best  balance 
work  force  capabilities  and  costs.  Thus,  in  addition  to  collecting  performance  data,  this  project 
contains  an  analytic  approach  that  integrates  performance  measurement  resuits  with  manpower, 
personnel,  and  training  (MPT)  and  economic  modeling  techniques.  The  resulting  analytic  structure 
and  data  can  be  used  to  predict  the  effects  of  changes  in  aptitude  requirements  in  three  critical 
areas: 

1.  the  average  growth  in  individuals’  task  proficiency; 

2.  the  overall  productive  capacity,  personnel  structure,  and  costs  of  the  specialty;  and 

3.  the  productive  capacity  of  individual  units  that  employ  a  mix  of  members  of  the  specialty. 

The  performance  data  for  the  TTP  project  were  collected  using  a  promising,  yet  fundamentally 
unique,  technique.  This  technique  uses  supervisors’  assessments  of  their  subordinates’  relative 
performance  within  homogeneous  sets  of  tasks  in  a  particuiar  speciaity.  This  data  collection 
method,  while  not  perfected,  has  the  potential  to  be  unobstrusive,  efficient,  and  comprehensive. 
Thus,  this  technique  represents  an  improvement  over  many  data  collection  and  performance 
assessment  techniques  that  are  currently  being  used  or  have  been  used  in  the  past. 

This  study  evaluated  only  one  specialty  (Avionic  Communications  Specialist,  AFS  328X0)  and 
concentrated  on  the  evaluation  of  first-term  airmen.  This  paper  describes  (a)  the  development 
of  the  TTP  approach  and  the  acquisition  of  data,  (b)  the  evaluation  of  the  TTP  measures  as 
compared  to  other  performance  measures,  (c)  the  formulation  of  a  model  which  is  used  to 
minimize  first-term  cost  per  productive  unit  through  the  selection  of  an  optimum  level  of  aptitude, 
and  (d)  the  solution  of  that  model. 
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II.  DEVELOPMENT  OF  THE  TTP  APPROACH 
AND  ACQUISITION  OF  DATA 


Development  and  Methodology 

Both  practical  needs  and  political  pressures  are  increasing  for  the  Air  Force  to  develop 
efficient  and  effective  ways  of; 

1.  evaluating  the  influence  of  mental  aptitudes  on  job  performance  in  individual  specialties, 
and 

2.  identifying  for  each  specialty  the  minimum  aptitude  requirements  that  would  optimally 
balance  its  work  force's  operational  capabilities  and  the  practicality  of  managing  that  work  force. 


One  of  the  most  crucial  practical  matters,  of  course,  is  the  recruitment  of  sufTicient  personnel 
with  the  requisite  aptitudes. 

There  has  been  much  research  conducted  on  a  variety  of  performance  measurement  methods 
(see  Landy  &  Farr,  1983).  This  includes  scores  from  written  tests  in  training  school, 
paper-and-pencll  job  knowledge  tests,  hands-on  work  sample  tests,  and  a  variety  of  behaviorally 
anchored  rating  scales. 

A  fundamentally  different  technique  now  appears  promising.  This  approach  uses  supervisors' 
assessments  of  their  subordinates'  relative  performance  within  homogeneous  sets  of  tasks.  It 
has  the  potential  to  be  less  obtrusive  than  hands-on  measures,  yet  efficient  and  comprehensive. 
The  data  collection  method  was  developed  originally  to  ascertain  the  effects  of  accumulated 
experience  on  task  performance,  in  order  to  identify  alternative  complements  of  personnel  that 
could  accomplish  specific  amounts  and  mixes  of  work  in  individual  work  centers. 

The  TTP  project  is  composed  of  a  sequence  of  three  research  and  development  (R&D) 
phases  that  are  based  on  differentiation  of  individuals'  productive  efficiency,  considering  both 
the  quantity  and  the  quality  of  job  performance.  Productive  efficiency  is  a  readily  interpreted 
concept  that  enables: 

1.  estimation  of  the  time  required  for  individuals  with  differing  mental  aptitudes  to  achieve 
proficiency  in  different  types  of  work, 

2.  trade-offs  between  an  occupation's  productive  capability  and  its  entrance  requirements, 
and 

3.  evaluation  of  the  effects  on  unit  performance  of  altering  entrance  requirements. 

in  practice,  the  approach  rests  on  subjective  data  provided  by  supervisors.  These  data  are 
collected  in  a  controlled  and  systematic  manner,  and  can  be  compared  with  objective  data 
collected  via  direct  observation. 
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During  Phase  1,  both  subjective  and  objective  data  are  collected  and  compared  for  selected 
tasks  within  a  single  Air  Force  specialty  (AFS).  This  comparison  serves  as  a  validation  of  the 
accuracy  of  the  subjective  data.  Both  the  subjective  approach  and  its  results  are  also  compared 
with  alternative  performance  measurement  methods  in  this  first  phase.  In  Phase  2,  an  analytic 
structure  is  developed  for  assessing,  in  an  integrated  manner,  the  effects  of  changing  an  AFS's 
aptitude  requirements  on  its  productive  potential,  career  structure,  and  MPT  costs.  In  Phase 
3,  performance  measurement  data  collected  in  Phase  1  are  integrated  into  the  model  developed 
in  Phase  2. 

The  prime  objectives  of  Phase  1  are  to  establish  the  validity  of  supervisors'  subjective 
evaluations  of  their  subordinates’  relative  productive  efficiencies,  to  validate  the  relationships 
these  evaluations  imply  between  job  performance  and  mental  aptitudes,  and  to  compare  this 
new  subjectively  based  approach  to  other  perfomnance  measurement  methods.  Since  the 
approach  requires  supervisors  to  evaluate  subordinates’  relative  performance  only  with  respect 
to  groups  of  tasks,  the  initial  study  uses  a  thoroughly  representative  set  of  task  groups.  That 
is,  insofar  as  possible,  the  task  groups,  for  which  both  objective  and  subjective  data  were 
collected,  span  the  range  of  work  performed  by  enlisted  personnel  (e.g.,  they  include  groups 
that  represent  fairly  simple  technical  tasks,  complex  technical  tasks,  clerical  work,  and  management 
tasks). 

Phase  2  develops  and  implements  (in  a  prototype  computer  code)  a  conceptual  model  for 
evaluating  the  effects  of  changing  the  minimum  mental  aptitudes  required  for  entry  into  an  AFS. 
This  model  uses  the  notions  of  productive  potential  and  time  to  proficiency,  and  can  be  used 
with  the  corresponding  data  obtained  through  the  data  collection  and  analysis  efforts  of  Phase 
1  or  some  other  mechanism  external  to  this  project. 

Phase  3  exercises  the  model  developed  in  Phase  2  using  the  data  and  relationships  developed 
in  Phase  1,  systematically  altering  the  requirements  for  AFS  entry  with  the  patterns  identified. 
Examples  of  questions  to  be  examined  include: 

1.  How  much  would  overall  productive  capability  and  costs  change  with  specified  aptitude 
requirement  changes? 

2.  How  much  would  the  average  time  required  to  achieve  proficiency  change? 

3.  How  much  difference  would  there  be  in  the  average  amount  of  fully  proficient  time  the 
USAF  would  realize  from  each  recruit? 

A  critical  component  of  the  TTP  project  is  assessing  the  relationship  between  individual 
characteristics  (most  notably  cognitive  ability  and  job  experience)  and  job  performance.  This 
requires  that  a  performance  measurement  procedure  be  developed  and  validated.  The  performance 
measurement  technique  which  will  be  employed  is  built  around  supervisors’  assessments  of 
subordinates’  performance  in  each  of  several  homogeneous  task  groups  drawn  from  the  job 
responsibilities  of  the  Avionic  Communications  Specialty  (AFS  328X0).  For  the  purpose  of  this 
study,  it  was  desired  that  the  developed  task  groupings  have  the  following  characteristics: 

V  1.  Each  group  should  be  composed  of  tasks  which  are  coperformed.  Since  supervisors 

will  be  rating  performance  on  the  task  groups  as  a  whole,  it  is  essential  that  the  group  consist 
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of  tasks  which  logically  relate  to  one  another  and  accordingly  can  be  viewed  as  a  coherent 
whole. 


2.  The  tasks  in  each  group  should  be  homogeneous  with  respect  to  difficulty.  It  is  likely 
that  tasks  that  require  little  ability  and  training  will  not  distinguish  low  from  high  ability  personnel 
since  this  ability  will  confer  no  advantage  in  task  mastery  or  performance.  Conversely,  since 
demanding  tasks  require  considerably  more  training  and  experience  for  mastery,  the  ability  to 
learn  arxJ  retain  complex  skills  will  play  a  more  important  role  In  job  performance.  Therefore, 
task  difficulty  is  likely  to  moderate  the  relationship  between  ability  and  experience  on  the  one 
hand  and  job  performance  on  the  other.  It  Is  therefore  important  that  each  task  group  sample 
a  relatively  narrow  range  on  the  task  difficulty  continuum  to  permit  an  unconfounded  estimate 
of  the  abiiity/performance  relationship. 

3.  The  task  groups  should  be  manageable  in  size  to  facilitate  the  rating  process.  The 
rating  task  which  the  supervisors  will  be  asked  to  perform  in  this  study  will  involve  a  fairly 
complex  judgment  in  which  the  performance  efficiencies  of  two  individuals  will  be  compared 
across  several  tasks  (and,  through  recall,  across  many  situations)  and  expressed  through  the 
assignment  of  a  single  rating.  If  the  task  groups  are  very  large  (e.g.,  consisting  of  50  tasks), 
it  will  be  difficult,  if  not  impossible,  for  the  rater  to  give  equal  consideration  and  weight  to  all 
tasks  In  the  group  while  performing  the  rating.  Only  those  tasks  which  the  rater  views  as  the 
most  important  or  most  representative  of  the  group  would  actually  be  considered  in  the  rating. 
Since  the  selection  of  a  subset  of  tasks  would  probably  vary  from  rater  to  rater,  this  would 
introduce  a  considerable  degree  of  error  variance  in  the  performance  measures.  Although  no 
single  number  of  tasks  can  be  firmly  established  and  unequivocally  defended  as  an  absolute 
criterion  of  task  group  size,  it  was  considered  desirable  to  limit  the  task  group  size  to  15  tasks 
or  less. 

4.  Taken  together,  the  task  groups  should  sample  as  broad  a  range  of  task  difficulty  levels 
as  possible.  Although  each  group  was.  developed  to  be  internally  homogeneous  with  respect 
to  task  difficulty,  efforts  were  taken  to  ensure  that  the  groups  were  distinct  from  one  another 
on  this  dimension.  The  between-group  variance  in  task  difficulty  allows  statistical  determination 
of  the  effect  of  this  variable  on  the  abiiity/performance  relationship. 

These  criteria  define  the  desired  features  of  the  task  groupings  to  be  established.  Though 
the  criteria  are  not  mutually  exclusive,  it  was  recognized  that  it  would  be  improbable  that  a 
set  of  task  groupings  would  be  defined  which  perfectly  met  all  of  them.  Trade-offs  would 
inevitably  arise  through  which  one  of  the  criteria  would  need  to  be  relaxed  in  order  to  achieve 
another.  In  making  these  trade-offs,  the  relative  priority  of  the  criteria  paralleled  the  order  in 
which  they  are  listed  above. 


Stepa  In  Defining  Ta»k  Groups 

The  development  of  the  task  groupings  occurred  in  a  series  of  stages  in  which  they  were 
first  established  with  regard  to  task  Coperformance  and  then  iteratively  refined  to  minimize 
within-group  variance  in  Task  Difficulty.  The  first  step  in  defining  the  task  groups  consisted  of 
a  cluster  analysis  of  all  tasks  In  the  Occupational  Research  Data  Bank  (ORDB)  for  the  328X0 
APS.  Only  data  from  respondents  with  between  1  to  48  months  of  service  were  used  in  this 
analysis,  consistent  with  the  focus  of  the  larger  study.  This  analysis,  performed  on  the  basis 
of  task  Coperformance,  produced  a  total  of  44  acceptable  task  clusters.  This  number  was 
reduced  to  10  by  eliminating  from  further  consideration  those  task  groups  in  Duty  Areas  A 


through  F  (i.e.,  maintenance  management,  administration,  and  training  tasks)  or  in  duty  areas 
performed  by  less  than  20%  of  the  first-term  personnel  in  this  AFS.  The  first  criterion  was 
appiied  to  be  consistent  with  this  study's  focus  on  production-iike  work;  the  second  sought  to 
eiiminate  low-density  task  areas  for  which  sufficient  individuai  performance  data  were  uniikely 
to  be  obtainable. 

Summary  statistics  on  the  task  clusters  defined  through  this  procedure  are  dispiayed  in 
Table  1.  As  shown,  two  of  the  clusters  (Task  Groups  1  and  4)  contained  considerabiy  more 
tasks  (52  and  34  tasks,  respectively)  than  any  of  the  others  and  considerably  more  than  were 
thought  desirable.  A  second  problem  was  in  the  within-cluster  variability  in  Task  Difficulty. 
Inasmuch  as  the  Task  Difficulty  is  scaled  to  have  a  standard  deviation  of  1.00,  the  three  task 
clusters  with  standard  deviations  greater  than  this  (Task  Groups  1,  2,  and  9)  are  less  homogeneous 
on  this  dimension,  than  the  overali  population  of  tasks.  The  task  ciusters  were  refined  to  correct 
these  problems. 


Table  1.  Summary  Statistics  on  Original  Task  Clusters 


Task 

#  Of 

Coperformance 

Task  difficulty 

group  # 

tasks 

within  subgroups 

Mean 

SD 

1 

52 

51.1 

4.67 

1.19 

2 

4 

36.2 

3.34 

1.12 

3 

5 

21.1 

4.34 

0.67 

4 

34 

64.7 

5.63 

0.89 

5 

5 

44.1 

4.92 

0.94 

6 

5 

36.9 

3.32 

0.66 

7 

17 

61.4 

5.14 

0.86 

8 

9 

30.6 

4.31 

0.49 

9 

9 

25.4 

3.10 

1.14 

The  large  task  clusters  were  first  reduced  in  size  by  decomposing  them  into  the  subgroups 
from  which  they  were  formed  during  the  cluster  analysis.  This  reduced  Task  Group  1  to  two 
groups  (designated  1A  and  IB  in  Table  2)  containing  26  tasks  each,  and  Task  Group  4  into 
three  groups  (designated  Task  Groups  4A,  46,  and  4C  in  Table  2)  consisting  of  9,  21,  and  4 
tasks,  respectively. 


Table  2.  Summary  Statistics  of  Second 
Stage  Task  Clusters 


Task 

#  Of 

Task  difficulty 

Group  # 

-  tasks 

Mean 

SIT 

1A 

26 

4.66 

1.22 

1B 

26 

4.68 

1.15 

4A 

9 

5.39 

0.84 

4B 

21 

5.78 

0.90 

4C 

4 

5.42 

0.81 

5 


Steps  were  next  taken  to  improve  the  groups’  homogeneity  in  terms  of  Task  Difficulty.  Since 
Groups  1A  and  IB  were  stilt  of  substantial  size,  they  were  further  subdivided  by  splitting  each 
at  their  median  Task  Difficulty  rating.  This  reduced  the  standard  deviations  of  all  task  groups 
derived  from  the  original  Task  Group  1  to  substantially  less  than  1.00. 

Futher  homogeneity  in  Task  Difficulty  was  achieved  by  eliminating  from  each  task  group 
any  task  which  was  an  outlier  in  either  Task  Difficulty  or  Coperformance.  This  was  done  by 
reviewing  scatterplots  of  the  tasks  in  each  task  group,  plotted  as  a  function  of  Task  Difficulty 
and  Coperformance  with  the  other  tasks  in  the  task  group.  On  the  basis  of  the  scatterpiot 
review,  three  tasks  were  eliminated  from  the  task  groups,  yielding  the  15  task  groups  summarized 
in  Table  3. 


Table  3.  Summary  Statistics  for  Task  Clusters 


Task 
group  # 

Task  difficulty 

Mean  SO 

#  Of 

tasks 

#  of 

WTPT  tasks 

1 

3.646 

0.340 

14 

4 

2 

5.858 

0.678 

12 

5 

3 

4.159 

0.438 

11 

2 

4 

5.542 

0.640 

13 

7 

5 

4.373 

0.574 

7 

0 

6 

5.391 

0.836 

9 

1 

7 

6.617 

0.477 

10 

2 

8 

5.010 

0.372 

11 

3 

9 

5.419 

0.813 

4 

0 

10 

4.919 

0.935 

5 

1 

11 

3.017 

0.291 

4 

0 

12 

5.794 

0.597 

7 

2 

13 

4.549 

0.657 

10 

0 

14 

4.313 

0.486 

9 

0 

15 

2.835 

0.923 

8 

0 

Finally,  the  task  groups  were  reviewed  by  subject-matter  experts  (SMEs),  in  this  case  328X0 
supervisory  personnel.  A  prime  concern  was  whether  the  task  groups  developed  were  truly 
distinct,  meaningful,  and  identifiable  components  of  the  328X0  job  responsibilities.  The  task 
groups  were  modified  in  light  of  the  SME  input  to  yield  the  final  10  groups  which  were  used 
in  the  TTP  questionnaire.  The  final  task  groups  are  contained  in  Appendix  A. 

Summary  data  indicate  that  the  task  groups,  in  general,  possess  the  desired  characteristics. 
Coperformance  among  each  group's  tasks  was  established  by  the  cluster  analysis  that  originally 
defined  the  groupings,  in  addition,  the  within-group  standard  deviations  in  task  difficulty  show 
that  the  task  groups  are  homogeneous,  with  no  group  having  a  standard  deviation  greater  than 
1.00,  and  most  having  standard  deviations  substantially  less.  Across  the  task  groups,  however, 
there  is  a  considerable  range  in  Task  Difficulty.  The  task  group  mean  Task  Difficulty  ratings 
vary  from  3.7  to  6.0.  This  variation  in  Task  Difficulty  was  sufficient  for  this  project’s  anaiytic 
requirements.  The  task  group  size  goals  were  also  achieved.  All  of  the  groups  are  composed 
of  less  than  15  tasks,  and  half  of  the  groups  have  less  than  10  tasks. 
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Though  not  an  explicit  criterion,  an  additional  desired  feature  was  that  the  tasks  in  the  task 
groups  of  the  TTP  Questionnaire  overlap  with  the  tasks  contained  in  the  Walk-Through  Performance 
.  Test  (WTPT).  This  allows  a  direct  comparison  of  hands-on  job  performance  information  collected 

in  the  WTPT  to  the  supervisory  ratings  of  job  performance  collected  with  the  TTP  methodology. 
Fortunately,  this  overlap  was  satisfactory.  Overall,  9  of  the  10  derived  task  groups  contain 
■4  WTPT  tasks,  and  a  total  of  25  tasks  overlap  between  the  two  data  collection  methodologies 

(see  Appendix  A). 


Questionnaire  Development 

After  the  task  groups  were  defined,  a  questionnaire  was  developed  to  collect  the  required 
information.  This  included  background  data  on  the  responding  supervisor  and  corresponding 
subordinates,  relative  time  required  to  perform  a  fixed  amount  of  work  at  a  specified  level  of 
qu2tlity  in  each  task  group,  the  relative  supervision  time  required  by  each  subordinate  for  task 
instruction  or  inspection,  and  how  often  each  subordinate  actually  works  on  a  task  in  each 
group.  The  response  grid  from  the  unpublished  Aerospace  Ground  Equipment  (AGE)  Performance 
Differences  Questionnaire  was  adapted  to  collect  the  data.  Each  task  group  was  given  its  own 
response  grid,  which  was  accompanied  by  a  list  of  benchmark  performance  times  for  each  task 
within  the  cluster.  The  benchmark  times'  were  determined  by  averaging  estimates  provided  by 
six  experienced  328X0  SMEs  for  the  typical  first-term  Avionic  Communications  Specialist. 

The  draft  questionnaire  was  then  reviewed  by  both  Air  Force  Human  Resources  Laboratory 
(AFHRL)  personnei  and  practicing  supervisors  in  the  appropriate  work  centers  prior  to  field  use. 
This  provided  opportunities  to  refine  instructions,  individual  questions,  and  response  formats, 
and  to  further  revise  task  groups  to  improve  within-group  homogeneity. 

The  questionnaire  was  then  pretested  by  having  each  part  completed  by  supervisors  at  three 
Air  Force  bases:  Seymour  Johnson,  Norton,  and  Beale.  The  supervisors  were  instructed  to 
make  judgments  in  a  systematic  manner  using  a  consistent  frame  of  reference.  Members  of 
the  research  team  were  present  to  observe  difficulties,  provide  assistance,  ascertain  whether 
the  forms  were  completed  correctly,  and  obtain  feedback  regarding  instructions,  difficulties,  and 
completion  time  requirements.  Based  on  the  pretest  results,  the  questionnaire  and  instructions 
were  revised.  At  this  point  the  revisions  consisted  chiefly  of  clarification  of  instructions.  The 
final  version  of  the  TTP  Questionnaire  is  contained  in  Appendix  B. 


III.  COMPARISON  OF  PERFORMANCE  MEASUREMENT  METHODS 


Air  Force  Job  Performance  Measurement  System  (JPMS) 

The  Job  Performance  Measurement  (JPM)  project  involves  the  developrffent  of  a  variety  of 
measurement  techniques  for  collecting  valid,  accurate,  and  reliable  hands-on  job  performance 
information.  This  comprehensive  approach  to  measuring  job  performance  is  based  on  the 
rationale  that  the  various  methods-while  to  some  extent  overlapping-measure  different  aspects 
of  the  criterion  space  with  differing  levels  of  accuracy  (Kavanagh,  Borman,  Hedge,  &  Gould, 
1987).  Less  expensive,  easier-to-administer  interview  tests  and  performance  ratings  will  then 
be  compared  and  evaluated  as  substitutes  for  the  more  expensive,  labor-intensive,  hands-on 
performance  measures.  A  brief  description  of  the  JPMS  follows.  A  more  thorough  description 
is  provided  in  Hedge  and  Teachout  (1986). 
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Walk-Through  Perfonnance  Testing  fWTPT) 


The  WTPT  is  founded  on  the  work  sample  philosophy  but  expands  the  testing  domain  by 
including  an  interview  testing  component  (Hedge,  1984).  The  hands-on  component  resembles 
a  traditional  hands-on  work  sample  test  designed  to  measure  task  proficiency.  The  interview 
component  is  conducted  at  the  work  site  in  a  "show  and  tell"  fashion  that  allows  the  incumbent 
to  "visually  and  verbally"  describe  how  a  step  should  be  accomplished.  For  each  component, 
a  test  administrator  uses  a  checklist  to  record  whether  the  steps  necessary  for  successful 
performance  on  a  task  are  correctly  performed  or  described.  A  5-point  overail  proficiency  rating 
is  recorded  by  the  administrator  after  the  completion  of  each  task.  Additional  information 
concerning  time  required  to  complete  each  task  and  specific  amounts  of  task  experience  required 
is  also  collected. 


Rating  Forms 

Rating  forms  range  in  detail  from  the  very  micro  to  the  very  macro  in  order  to  provide  a 
thorough  coverage  of  the  performance  evaluation  continuum.  Four  rating  forms-task,  dimensional, 
global,  and  Air  Force-wide-are  completed  by  three  separate  rating  sources  (supervisor,  peer, 
and  self).  All  forms  employ  a  5-point  adjectivally  anchored  rating  scale,  and  specific  behavioral 
descriptions  are  included  to  provide  detailed  information  to  assist  raters  in  making  accurate 
judgments. 

Task  Rating  Form.  The  task  rating  form  provides  the  most  specific  rating  data.  This  form 
contains  approximately  40  Avionic  Communications  Specialist  tasks,  including  all  those  in  the 
WTPT. 

Dimensional  Rating  Form.  The  dimensional  rating  form  provides  the  second  most  specific 
rating  data.  Technical  proficiency  is  rated  across  important  areas  of  the  job.  For  AFS  328X0 
these  areas  include:  Avionics  Administrative  Functions,  Troubleshooting,  Remove/Replace,  Bench 
Check/Operational  Check,  and  General  Avionics  Maintenance. 

Global  Rating  Form.  The  global  rating  form  is  designed  to  collect  overall  ratings  of 
proficiency.  Two  items  are  used  to  assess  the  technical  proficiency  and  interpersonal  proficiency 
of  job  incumbents. 

Air  Force-Wide  Rating  Form.  The  Air  Force-Wide  rating  form  is  designed  to  be  representative 
of  all  specialties  in  the  Air  Force.  The  form  focuses  on  eight  performance  factors  important 
for  all  first-term  Air  Force  personnel.  These  factors  consist  of:  Technical  Knowledge/Skill, 
Initiative/Effort,  Knowledge  of  and  Adherence  to  Regulations/Orders,  Integrity,  Leadership,  Military 
Appearance,  Seif-Development,  and  Self-Control. 

In  sum,  the  WTPT  and  rating  forms  constitute  the  JPMS.  This  multiple  measurement  approach 
allows  the  identification  of  a  measure  or  combinations  of  measures  to  be  used  as  a  criterion 
in  selection,  classification,  and  training  R&O.  In  addition,  surrogate  measures  can  be  identified 
and  substituted  for  the  more  expensive  and  time-consuming  hands-on  procedure. 
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Comparing  TTP  to  the  Job  Performance  Measurement  System 


The  TTP  Questionnaire  was  completed  by  supervisors  who  indicated  the  relative  performance 
and  supervisory  times  required  by  each  of  their  subordinates  for  10  task  clusters.  The  task 
clusters  were  constructed  to  form  logical  groupings  of  tasks  which  are  homogeneous  with 
respect  to  Task  Difficulty  and  have  a  relatively  high  rate  of  Coperformance  within  the  cluster 
as  described  in  Section  II  of  this  paper. 

The  supervisor  was  provided  with  benchmark  performance  time  for  each  task  within  each 
cluster.  The  supervisor  then  chose  a  benchmark  worker  who  performed  closest  to  the  pace 
represented  by  the  benchmark  times  for  each  cluster.  The  benchmark  worker  concept  was 
used  to  contrd  for  differences  In  evaluation  across  supervisors.  The  supervisor  then  estimated 
how  long  it  would  take  each  subordinate  to  comfi^ete  the  amount  of  work  that  the  benchmark 
worker  could  perform  in  1  hour.  The  supervisor  also  estimated  the  supervision  time  required 
for  each  worker  to  complete  that  amount  of  work. 

Data  Collection.  The  TTP  Questionnaire  was  administered  concurrently  with  the  JPMS  at 
18  Air  Force  Bases.  These  bases  are  listed  in  Appendix  C. 

The  questionnaire  was  completed  by  18  supervisors  for  83  job  incumbents  in  AFS  328X0. 
Of  these,  58  were  matched  with  the  JPMS  data  file  for  AFS  328X0. .  A  full  set  of  data  was  not 
available  for  some  of  the  58  matched  cases.  For  some  cases,  fewer  than  10  TTP  task  clusters 
were  completed  or  data  from  one  or  more  WTPT  tasks  were  missing.  The  number  of  complete 
cases  available  for  comparison  varied  by  task.  There  were  as  many  as  58  and  as  few  as  5 
pairwise  complete  cases. 


TTP  Versus  Global  Ratings 

The  TTP  Questionnaire  was  designed  to  measure  the  relative  rate  at  which  individuals  in  an 
AFS  perform  tasks  within  selected  task  clusters  and  the  time  required  to  supervise  that 
performance.  The  global  technical  proficiency  rating  measures  the  extent  to  which  the  individual 
meets  or  exceeds  an  acceptable  level  of  technical  proficiency  over  all  the  tasks  he  or  she 
performs.  Since  higher  levels  of  proficiency  are  reflected  in  faster  performance  times,  aggregated 
TTP  performance  and  supervision  times  should  be  negatively  correlated  with  global  technical 
proficiency.  The  TTP  Questionnaire  was  not  designed  to  measure  interpersonal  proficiency. 
Any  observed  correlation  between  the  TTP  measures  and  global  interpersonal  proficiency  would 
be  spurious.  In  summary,  it  is  hypothesized  that  the  TTP  measures  are  negatively  correlated 
with  technical  proficiency  ratings  and  uncorrelated  with  (independent  of)  interpersonal  proficiency 
ratings. 

Table  4  displays  the  correlation  coefficients  for  comparing  the  TTP  measures  to  the  global 
ratings.  The  TTP  times  are  weighted  averages  of  the  10  performance  or  supervision  times. 
The  average  percent  time  spent  by  first-term  328X0s  in  each  task  cluster  was  obtained  from 
the  Comprehensive  Occupational  Data  Analysis  Programs  (CODAP)  data  base  and  used  to 
weight  the  averages.  Both  of  the  global  ratings  were  tested  for  correlation  with  the  aggregate 
performance  and  supervision  times  from  the  TTP  questionnaire.  A  significant  correlation  with 
the  TTP  measures  was  observed  for  the  technical  proficiency  ratings.  A  significant  correlation 
was  not  observed  when  comparing  the  TTP  measures  to  the  interpersonal  proficiency  ratings. 
These  findings  support  the  above  hypotheses  by  confirming  the  relationship  between  the  TTP 
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msesures  and  technical  proficiency  while  showing  no  statistical  correlatioii  with  interpersonal 
proficiency. 


Table  4.  Correlation  Coefficients  for  TTP  Performance  and 
Supervision  Times  by  Global  Ratings 


Global  rating 

Performance 

time 

Supervision 

time 

Supervisor  Technical  Proficiency 

-0.5246** 

-0.4232** 

Peer  Technical  Proficiency 

-0.4064** 

-0.4523** 

Self  Technical  Proficiency 

-0.4997** 

-0.4331** 

Supervisor  Interpersonal  Proficiency 

-0.1851 

-0.1179 

Peer  Interpersonal  Proficiency 

-0.1739 

-0.1959 

Self  Interpersonal  Proficiency 

-0.0687 

-0.0079 

Note.  Based  on  58  observations. 
^Significant  at  the  0.01  level. 
^^Significant  at  the  0.001  level. 


TTP  Versus  Dimensional  Ratings 


The  TTP  task  clusters  span  the  range  of  the  dimensional  ratings.  Again,  since  higher  ratings 
are  expected  to  be  associated  with  faster  performance  times  and  less  supervision,  aggregated 
TTP  measures  should  be  negatively  correlated  with  the  five  dimensional  ratings  as  well  as  with 
the  average  of  the  five  ratings.  Table  5  displays  the  correlation  coefficients  for  the  TTP  measures 
compared  to  the  dimensional  ratings  for  self,  supervisor,  and  peer.  The  supervisor  and  peer 
ratings  showed  statistically  significant  correlations  with  the  TTP  measures  for  all  five  dimensions. 
The  self  ratings  were  significantly  correlated  with  four  of  the  five  dimensions;  however,  no 
statistical  significance  was  found  for  correlations  between  either  TTP  measure  and  Avionics 
Administrative  Functions  nor  between  TTP  supervision  time  and  Remove/Replace  for  the  self 
ratings.  Thus,  the  TTP  measures  show  a  moderate  correlation  with  the  majority  of  the  dimensional 
ratings. 


TTP  Versus  Air  Force-Wide  Ratings 

The  TTP  measures  were  expected  to  be  negatively  correlated  with  the  Technical  Knowiedge/Skill 
component  of  the  Air  Force-Wide  Rating  Form  and  uncorrelated  with  the  other  Air  Force-Wide 
components.  This  was  expected  because  higher  levels  of  technical  knowledge  and  skill  should 
be  associated  with  faster  performance  time  and  less  supervision.  Table  6  displays  the  correlation 
coefficients  for  the  TTP  measures  compared  to  the  eight  Air  Force-Wide  ratings.  As  expected, 
the  Technical  Knowledge/Skill  component  was  significantly  correlated  with  both  TTP  measures 
across  all  raters.  Supervisor  and  peer  Leadership  ratings  were  also  significantly  correlated  with 
the  TTP  measures,  as  was  the  supervisor  rating  of  Integrity.  The  Self-Development  component 
was  significantly  correlated  with  both  TTP  measures  for  the  self  and  peer  ratings  and  with 
supervision  time  for  the  supervisor  ratings.  A  few  other  pairs  of  measures  were  significantly 
correlated  as  indicated  in  the  table. 
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Table  5.  Correlation  Coefficients  for  Dimensional  Ratings  Versus 
Aggregate  TTP  Performance  and  Supervision  Times 
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Table  6.  Correlation  Coefficients  for  Air  Force-Wide  Rating  Form  Versus 
Aggregate  TTP  Performance  and  Supervision  Times 
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Significant  at  the  0.05  level. 
Significant  at  the  0.01  level. 


TTP  Versus  Task  Ratings 


The  TTP  measures  for  each  task  cluster  should  be  negatively  correlated  with  the  task  ratings 
for  that  cluster.  This  was  expected  because  a  high  task  rating  should  be  associated  with  a 
low  performance  time  and  a  low  supervision  time.  Table  7  gives  the  correlation  for  the  TTP 
measures  compared  to  aggregated  self,  supervisor,  and  peer  task  ratings  within  each  task 
cluster.  In  Cluster  1,  both  TTP  times  were  significantly  correlated  with  the  peer  ratings,  as 
was  supervision  time  with  the  supervisor  ratings,  in  Cluster  2,  the  TTP  measures  were  significantly 
correlated  with  the  supervisor  and  peer  ratings.  TTP  performance  time  was  significantly  correlated 
with  the  supervisor  and  peer  ratings  in  Cluster  3.  No  significant  correlations  were  observed 
for  Cluster  4.  in  Cluster  5,  both  TTP  measures  were  significantly  correlated  with  the  peer 
ratings,  as  was  performance  time  with  the  supervisor  ratings.  Only  supervision  time  and  the 
supervisor  ratings  were  significantly  correlated  in  Cluster  6.  Only  performance  time  and  the 
peer  ratings  were  significantly  correlated  in  Cluster  7.  The  peer  ratings  for  Cluster  8  were 
significantly  correlated  with  both  TTP  measures.  No  task  ratings  were  obtained  for  Cluster  9. 
in  Cluster  10,  both  TTP  measures  were  significantly  correlated  with  the  supervisor  ratings,  as 
was  performance  time  with  the  peer  ratings.  The  self  rating  was  not  significantly  correlated 
with  either  TTP  measure  in  any  cluster. 


TTP  Versus  Walk-Through  Performance  Tests 


Appendix  A  displays  the  assignment  of  specific  tasks  to  TTP  task  clusters.  The  tasks  which 
correspond  to  WTPT  tasks  for  which  data  were  available  are  indicated  by  an  asterisk  (*).  A 
total  of  25  of  the  TTP  tasks  were  common  to  the  WTPT.  At  least  one  WTPT  task  was  included 
in  all  but  one  of  the  TTP  clusters.  Each  cluster  contained  some  tasks  which  were  not  among 
the  WTPT  tasks. 

An  aggregate  WTPT  score  (measured  by  number  of  steps  correctly  completed),  overall  rating, 
and  performance  time  were  computed  for  each  of  the  10  TTP  task  clusters.  The  number  of 
steps  correctly  completed  was  summed  over  all  tasks  in  the  cluster.  To  adjust  for  WTPT 
differences  in  testing  (i.e..  Phase  2,  or  major  command-specific  tests),  the  sum  was  divided  by 
the  total  number  of  steps  on  which  the  individual  was  tested  for  that  cluster.  This  ratio  was 
used  as  a  cluster-level  measure  of  WTPT  performance.  When  both  interview  and  hands-on 
overall  ratings  were  available,  these  ratings  were  averaged  to  obtain  an  overall  rating  for  the 
task.  Otherwise,  whichever  rating  was  available  was  used  as  the  overall  rating  for  the  task. 
Interview  time  was  believed  to  be  inappropriate  because  the  time  required  to  complete  an 
interview  is  not  expected  to  be  related  to  the  time  required  to  actually  complete  a  task.  Thus 
the  hands-on  time,  when  available,  was  used  as  the  overall  time  for  each  task.  Cluster-level 
WTPT  ratings  were  computed  by  averaging  the  overall  WTPT  ratings  for  each  task  in  the  cluster, 
weighting  by  average  percent  time  spent  on  each  task.  These  weights  were  obtained  from  the 
CODAP  data  base  as  mentioned  previously.  This  weighting  of  tasks  by  average  percent  time 
spent  simulates  how  a  supervisor  might  form  a  composite  rating  for  a  cluster  in  the  TTP 
Questionnaire.  It  is  expected  that  a  task  on  which  the  average  individual  spends  a  greater 
proportion  of  his  or  her  time  would  have  a  stronger  influence  on  the  supervisor’s  composite 
assessment  than  tasks  which  make  up  a  smaller  proportion  of  the  average  individual's  productive 
time.  Similarly,  cluster-level  WTPT  hands-on  performance  times  were  computed  using  weighted 
averages. 
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Table  7.  Correlation  Coefficients  for  Task  Ratings  (Self,  Supervisor,  and  Peer) 
Versus  TTP  Performance  and  Supervision  Times 
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Significant  at  the  0.05  level. 
Significant  at  the  0.01  level. 


The  TTP  Questionnaire  coliected  only  cluster-level  data.  Thus,  direct  task-level  correlation 
between  TTP  and  WTPT  could  not  be  examined.  However  each  TTP  cluster  could  be  compared 
to  each  WTPT  task  within  that  cluster.  In  order  to  compare  TTP  and  WTPT  at  the  same  level 
of  aggregation,  WTPT  measures  were  aggregated  to  the  cluster  level.  Correlation  at  this  level 
was  expected  to  be  strongest  when  a  large  proportion  of  the  TTP  cluster  is  represented  by 
WTPT  tasks.  At  the  task  level,  correlations  of  each  TTP  cluster  with  each  WTPT  task  in  that 
cluster  were  examined.  At  the  aggregate  level,  correlations  between  TTP  clusters  and  WTPT 
measures  aggregated  to  the  cluster  level  were  examined.  Since  each  WTPT  task  represents 
only  a  small  portion  of  the  TTP  cluster,  failure  to  find  significant  task-level  correlation  may  be 
due  to  masking  effects  by  other  tasks  (TTP  and  WTPT)  in  the  cluster.  Aggregation  of  WTPT 
measures  to  the  cluster  level  could  cause  weaker  correlation  if  the  tasks  to  be  combined  are 
negatively  corretated.  However,  this  was  not  expected  since  the  clusters  were  constructed  to 
maximize  homogeneity  within  clusters. 

Table  8  displays  the  correlation  coefficients  for  the  TTP  performance  and  supervision  times 
with  task-level  and  cluster-level  WTPT  measures.  A  significant  correlation  was  found  between 
the  TTP  performance  time  for  Cluster  1  and  the  WTPT  hands-on  time  for  the  only  WTPT  task 
in  the  cluster.  For  Cluster  2,  significant  correlations  were  observed  for  both  TTP  performance 
time  and  supervision  time  when  compared  to  the  interview  performance  and  rating  for  the  tasks: 
"Safety-Wire  or  Bond  System  Components"  and  “Inspect  Parts  Received  from  Supply  or 
Manufacturer."  For  Cluster  3,  significant  correlations  were  observed  for  several  individual  tasks 
as  well  as  at  the  cluster  level.  No  significant  correlations  were  observed  for  Cluster  4.  For 
Cluster  5,  significant  correlations  were  observed  for  the  hands-on  time  for  several  tasks  as  well 
as  at  the  cluster  level.  A  significant  correlation  was  observed  between  the  TTP  performance 
time  and  the  interview  performance  and  rating  for  the  only  WTPT  task  in  Cluster  6.  The  TTP 
performance  time  was  significantly  correlated  with  the  hands-on  time  for  the  only  WTPT  task 
in  Cluster  7.  For  Cluster  8,  TTP  performance  time  and  the  hands-on  rating  for  the  task  "Isolate 
Malfunctions  in  HF  Receiver-Transmitters"  were  significantly  correlated.  Cluster  9  contained  no 
WTPT  tasks.  No  significant  correlations  were  observed  for  Cluster  10. 

The  TTP  measures  were  estimated  to  have  negative  correlation  coefficients  when  paired  with 
the  interview  and  hands-on  performance  and  ratings,  and  positive  correlation  coefficients  when 
paired  with  hands-on  time.  TTP  performance  and  supervision  time  for  Clusters  3  and  5,  in 
which  over  half  of  the  TTP  tasks  are  represented  by  WTPT,  show  significant  correlation  with 
aggregate  WTPT  performance  time.  This  supports  the  expectation  that  clusters  with  greater 
WTPT  representation  show  stronger  correlation.  A  control  cluster  in  which  all  tasks  are 
represented  by  WTPT  would  be  useful  to  verify  this  assumption.  Although  the  individual  task 
correlation  analysis  does  not  convincingly  verify  the  TTP  measures  as  a  surrogate  for  hands-on 
measures,  the  TTP  measures  offer  some  promise.  Further  analysis  on  a  larger  sample  with 
more  than  one  AFS  is  necessary  to  verify  the  relationship  between  TTP  and  hands-on  performance. 


Summary  of  Comparisons 

In  summary,  the  TTP  measurement  system  is  a  unique  approach  in  assessing  the  performance 
of  workers.  While  relationships  with  hands-on  measures  and  others  JPMS  measures  are  moderate, 
the  TTP  approach  has  potential  for  generating  modeling  data  to  use  in  estimating  the  effects 
of  changing  aptitude  requirements  on  productive  capacity. 
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Table  8.  Correlation  Coefficients  for  TTP  Performance  and 
Supervision  Times  Versus  WTPT  Task  Measures 


TTP  time 

Task _ WTPT  measure _ N  Performance  Supervision 


Task  Cluster  1  •  Maintenance  Administration  and  inspection 


Make  Entries  on  Maintenance 

Interview  Performance 

58 

-0.0628 

-0.0161 

Data  Collection  Record 

Interview  Rating 

58 

-0.0741 

-0.1740 

Forms.  (AFTO  Forms  349) 

Hands-on  Performance 

-0.0532 

-0.0749 

Hands-on  Rating 

58 

-0.1158 

0.0657 

Hands-on  Time 

58 

0.2909* 

0.1474 

Aggregate  Task  Cluster  1 

Performance 

58 

-0.0709 

-0.0758 

Rating 

58 

-0.1393 

-0.0585 

Time 

58 

0.2909* 

0.1474 

Task  Cluster  2  -  Fiightiine  Maintenance 

Safety-Wire  or 

Interview  Performance 

58 

-0.4106** 

-0.3385** 

Bond  System 

Interview  Rating 

58 

-0.3461** 

-0.3031* 

Components 

Hands-on  Performance 

58 

0.0321 

0.0506 

Hands-on  Rating 

58 

-0.1055 

-0.0404 

Hands-on  Time 

58 

-0.0106 

0.1916 

Remove  or  Replace  UHF 

Interview  Performance 

58 

0.0341 

-0.0236 

Receiver-T  ransmitters 

Interview  Rating 

58 

-0.0779 

-0.0399 

Remove  or  Replace  Inter¬ 

Interview  Performance 

22 

-0.0533 

-0.0986 

phone  Station  Control  Units 

Interview  Rating 

22 

-0.1360 

-0.2279 

Inspect  Parts  Received  from 

Interview  Performance 

58 

-0.3888** 

-0.3039* 

Supply  or  Manufacturer 

Interview  Rating 

58 

-0.2847* 

-0.2737* 

Aggregate  Task  Cluster  2 

Performance 

58 

-0.3721** 

-0.3035* 

Rating 

58 

-0.2469 

-0.2123 

Time 

58 

-0.0106 

0.1916 
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Table  8.  (Continued) 


TTP 

time 

Task 

WTPT  measure 

N  Performance 

Supervision 

Task  Cluster  3  -  Flightline  Troubleshooting 


Repair  Avionic  System 

Hands-on  Performance 

9 

0.2978 

-0.1032 

Wiring  or  Cables 

Hands-on  Rating 

9 

-0.1919 

-0.2521 

Hands-on  Time 

9 

0.0475 

0.1520 

Remove  or  Replace 

Hands-on  Performance 

55 

-0.3959** 

-0.3737’ 

Radio  Frequency  (RF) 

Hands-on  Rating 

55 

-0.2991* 

-0.2798' 

Coaxial  Connectors 

Hands-on  Time 

54 

0.2607 

0.3544’ 

Isolate  Malfunctions  in 

Interview  Performance 

55 

-0.1977 

-0.1240 

UHF  Systems 

Interview  Rating 

55 

-0.1152 

0.0022 

Trace  Circuits  or  Signals 

Hands-on  Performance 

55 

-0.3321* 

-0.1981 

Using  Wiring  Diagrams  or 

Hands-on  Rating 

55 

-0.2723* 

-0.1532 

Schematics 

Hands-on  Time 

55 

0.2591 

0.3031’ 

Remove  or  Replace 

Hands-on  Performance 

9 

-0.7021* 

0.0226 

Multiple  Wire  Plugs 

Hands-on  Rating 

9 

•0.5289 

0.0092 

Hands-on  Time 

9 

0.6213 

0.0726 

Aggregate  Task  Cluster  3 

Performance 

55 

-0.4448** 

-0.0261 

Rating 

55 

-0.3431* 

-0.2042 

Time 

55 

0.3257* 

0.4276’ 

Task  Cluster  4  -  Shop  Maintenance 


Isolate  Malfunctions  in 

Interview  Performance 

51 

-0.0658 

-0.1087 

Interphone  Cords 

Interview  Rating 

51 

-0.1084 

-0.1736 

Isolate  Malfunctions  in 

interview  Performance 

22 

-0.0644 

0.1693 

Interphone  Monitor 

Control  Units 

Interview  Rating 

22 

-0.0876 

0.0464 

Aggregate  Task  Cluster  4 

Performance 

51 

-0.1062 

-0.0604 

Rating 

51 

-0.1116 

-0.1584 
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Table  8.  (Continued) 


TTP 

time 

Task 

WTPT  measure 

N  Performance 

Supervision 

Task  Cluster  5  -  UHF  Shop  Maintenance 


Bench  Check  UHF 

Hands-on  Performance  .. 

49 

-0.3327* 

-0.2805 

Receiver-T  ransmitters 

Hands-on  Rating 

49 

-0.2373 

-0.2787 

Hands-on  Time 

49 

0.3361* 

0.3252* 

isolate  Malfunctions  in  UHF 

Interview  Performance 

49 

-0.1768 

-0.0234 

Receiver-T  ransmitters 

Interview  Rating 

49 

-0.2180 

-0.1584 

Bench  Check  UHF 

Hands-on  Performance 

5 

-0.4226 

0.2182 

Control  Units 

Hands-on  Rating 

5 

0.7906 

0.4082 

Hands-on  Time 

5 

0.7946 

-0.2052 

Remove  or  Replace  UHF 

interview  Performance 

49 

-0.2516 

-0.3063* 

Receiver-T  ransmitter 

Interview  Rating 

49 

-0.1380 

-0.2304 

Components 

Hands-on  Performance 

-0.1398 

-0.1175 

Hands-on  Rating 

49 

-0.2541 

-0.2805 

Hands-on  Time 

49 

0.3441* 

0.2788 

Set  Up  UHF  System- 

Hands-on  Performance 

0.1026 

0.0753 

Peculiar  Test  Equipment 

Hands-on  Rating 

49 

0.0584 

0.0376 

Hands-on  Time 

49 

0.2868* 

0.2394 

Perform  Time  Compliance 

Interview  Performance 

5 

0.3101 

-0.4804 

Technical  Order  (TCTO) 
Modifications  on  Avionic 
Systems 

Interview  Rating 

5 

0.0000 

-0.5601 

Aggregate  Task  Cluster  5 

Performance 

49 

-0.2671 

-0.2022 

Rating 

49 

-0.2466 

-0.2657 

Time 

49 

0.4249** 

0.3815** 

Task  Cluster  6  -  Flightline  HF  Maintenance 

Remove  or  Replace  HF  Interview  Performance 

25 

-0.5413** 

-0.0678 

Receiver-Transmitters  Interview  Rating 

25 

-0.4276* 

-0.1073 

Aggregate  Task  Cluster  6  Performance 

25 

-0.5413** 

-0.0678 

Rating 

25 

-0.4276* 

-0.1073 
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TTP  time 

Task _ WTPT  measure _ N  Performance  Supervision 


Task  Cluster  7  -  HF  Control  Box  In-Shop  Maintenance 


Bench  Check  HF 

Hand-on  Performance 

22 

0.0273 

0.3106 

Control  Boxes 

Hands-on  Rating 

22 

-0.1788 

0.0554 

Hands-on  Time 

22 

0.4484* 

0.3410 

Aggregate  Task  Cluster  7 

Performance 

22 

0.0273 

0.3106 

Rating 

22 

-0.1788 

-0.0554 

Time 

22 

0.4484* 

0.3410 

Task  Cluster  8  -  Receiver-Transmitter/HF  Coupler  Shop  Maintenance 


Remove  or  Replace  HF 

Interview  Performance 

22 

-0.2434 

-0.0763 

Receiver-ZT  ransmitter 

Interview  Rating 

21 

-0.2928 

-0.0725 

Subassemblies 

Hands-on  Performance 

22 

-0.2056 

-0.1771 

Hands-on  Rating 

22 

-0.3784 

-0.2610 

Hands-on  Time 

22 

-0.1646 

-0.3329 

Set  Up  HF  System- 

Hands-on  Performance 

40 

-0.0190 

-0.1186 

Peculiar  Test  Equipment 

Hands-on  Rating 

40 

-0.0114 

-0.1169 

Hands-on  Time 

40 

-0.2262 

0.0406 

Isolate  Malfunctions  in  HF 

Hands-on  Performance 

22 

-0.1672 

-0.1101 

Receiver-T  ransmitters 

Hands-on  Rating 

19 

-0.5659* 

0.3901 

Hands-on  Time 

19 

-0.3221 

-0.3299 

Aggregate  Task  Cluster  8 

Performance 

40 

-0.0973 

-0.1768 

Rating 

40 

-0.0386 

-0.1922 

Time 

40 

-0.1493 

0.0662 
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Table  8.  (Concluded) 


TTP 

time 

Task 

WTPT  measure 

N  Performance 

Supervision 

Task  Cluster  10  *  Shop  VHP  Maintenance 


Align  VHP  AM 

Hands-on  Performance 

19 

-0.2032 

-0.1461 

Receiver-T  ransmitters 

Hands-on  Rating 

19 

-0.0829 

-0.0429 

Hands-on  Time 

19 

0.1221 

0.1520 

Bench  Check  VHP  AM 

Hands-on  Performance 

19 

-0.1749 

-0.2417 

Receiver-T  ransmitters 

Hands-on  Rating 

19 

-0.1397 

-0.1957 

Hands-on  Time 

19 

0.2504 

0.2989 

Aggregate  Task  Cluster  10 

Performance 

19 

-0.2202 

-0.1745 

Rating 

19 

-0.1258 

-0.1246 

Time 

19 

0.1683 

0.2065 

Note.  N  -  Number  of  Observations. 
*Significant  at  the  0.05  level. 
**Significant  at  the  0.01  level. 


IV.  TIME  TO  PROFICIENCY  MODEL  DEVELOPMENT 

Overview  of  the  TTP  Model 


The  objective  of  the  TTP  model  is  to  identify  an  optimum  method  of  selecting  airmen  for 
entry  into  an  APS  which  minimizes  cost  per  productive  unit.  The  model  uses  the  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  electronics  score,  a  measure  of  aptitude,  as  a 
selection  variable.  The  model  is  used  to  find  the  electronics  score  requirement  associated  with 
the  smallest  cost  per  productive  unit  over  the  first  term.  The  model  consists  of  four  components: 


1.  Productive  Capacity  Model 

2.  Attrition  Model 

3.  Cost  Model 

4.  Distribution  of  Aptitudes 


The  productive  capacity  model  represents  productivity  as  a  function  of  experience  and 
aptitude.  It  is  generally  hypothesized  that  airmen  with  higher  aptitudes  are  more  productive 
than  are  airmen  with  lower  aptitudes,  and  that  productivity  increases  with  experience.  The 
attrition  model  represents  the  likelihood  of  remaining  in  service  for  each  month  in  the  first  term 
as  a  function  of  aptitude.  The  data  show  that  airmen  with  lower  aptitude  are  more  likely  to 
leave  the  Air  Force  early  than  those  with  higher  aptitude.  The  cost  model  represents  monthly 
training  costs  and  military  pay  through  the  first  term  and  includes  the  cost  of  recruiting.  Ideally, 
the  cost  model  would  be  broken  down  by  aptitude.  Unfortunately  this  level  of  discrimination 
could  not  be  obtained  at  this  time.  The  cost  model  is  therefore  independent  of  aptitude.  The 


dstribution  of  aptitude  is  a  modei  of  the  probability  density  function  (PDF)  for  Electronics 
Aptitude  Index  (Al)  scores  for  the  population  of  airmen.  From  this  model,  the  PDFs  for 
subpopulations  defined  by  selecting  airmen  on  the  basis  of  electronics  score  are  computed. 

An  objective  modei  is  derived  by  examining  the  functional  and  logical  relationships  of  the 
four  components  (production,  attrition,  cost,  and  aptitude).  Production  and  cost  are  accumulated 
over  the  first  term  and  adjusted  for  attrition  to  yield  expected  first-term  productive  capacity  and 
eqsected  first-term  cost.  The  ratio  of  expected  cost  to  expected  productive  capacity  is  then 
computed  for  each  of  five  groups  of  airmen  defined  by  electronics  score.  The  distribution  of 
electronics  scores  among  airmen  is  then  used  to  identify  the  subpopulation  with  the  smallest 
average  cost  per  productive  unit  for  the  first  term. 


Productive  Capacity  Model 

A  mathematical  model  can  be  formulated  to  express  productive  capacity  (inverse  (In)  of 
performance  time)  as  a  function  of  experience  (time  in  AFS)  and  mental  aptitude  (electronics 
score).  Research  has  shown  that  learning  curves  often  follow  S-shaped  functions  such  as 
y=  1/(1 -(-exp(-x)).  This  is  known  as  the  logistic  function.  The  domain  of  this  function  is  the 
interval  (0,ao).  The  range  is  (0,1).  Our  model  for  productive  capacity  is; 

P  =  1/(1 -(-exp  (-bo  -  biXi  -  baXa)) 

Where  P  =  productive  capacity 

Xi  =  experience  (months  in  AFS) 

X2  >=  mental  aptitude  index  (ASVAB  electronics  score) 
bo,  bi,  ba  are  parameters  to  be  estimated. 

To  ensure  that  productive  capacity,  P,  is  contained  in  the  interval  (0,1),  the  performance 
time  must  be  normalized.  The  productive  capacity,  P,  is  defined  as  t*/t  where  t  is  the  TTP 
performance  time  in  minutes  and  t*  is  1  minute  iess  than  the  minimum  observed  performance 
time  for  the  respective  cluster  (t*  =  min(t)  -  1).  Thus  t*  is  an  estimate  of  the  fastest  possible 
performance  time.  If  t*  is  the  smallest  possible  time  interval  in  which  a  defined  unit  of  work 
can  be  completed  at  an  acceptable  quality,  it  follows  that  1A*  is  the  maximum  number  of  work 
units  that  can  be  completed  per  unit  of  time.  For  any  individual  worker,  t  is  the  time  required 
for  that  worker  to  complete  the  defined  unit  of  work  at  an  acceptable  quality,  and  1/t  is  the 
number  of  work  units  that  the  individual  can  complete  per  unit  of  time.  Thus,  P  =  t*  /t  = 
(1/t)/(1/t*)  is  the  proportion  of  maximum  productivity  that  the  individual  has  obtained.  For 
example,  P  -  0.75  indicates  that  the  individual  performs  at  a  rate  which  is  75%  of  the  maximum 
achievable  productivity.  Note  that  when  t  is  very  close  to  t*,  P  approaches  1.0  (maximum 
productivity);  and  when  t  is  much  larger  than  t*,  P  approaches  zero  (no  productivity). 

The  next  objective  is  the  estimation  of  bi,  bz,  and  bo-  The  logistic  function  is  nonlinear 
with  respect  to  bo,  bi,  and  b2.  Algebraic  manipulation  and  a  log  transformation  yield  the 
following  function: 

In  (P  /  (1-P))  =  bo  +  biXi  +  b2X2. 

The  parameters  bi,  b2,  and  bo  can  now  be  estimated  using  multiple  linear  regression  with 
In  (P/(1-P))  as  the  dependent  variable  and  experience  and  electronics  score  as  explanatory 
variables.  Table  9  lists  the  estimated  parameters  bi,  b2,  and  bo  for  the  above  regression  model 
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Table  9.  Regression  Models  for  TTP  Performance  Time  as  a  Function  of  Experience  and  Aptitude 
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n  =  Number  of  Observations. 

R-Squared  =  Coefficient  of  Determination. 
P  =  Significance  Level. 

NS  =  Nonsignificant. 


for  each  of  the  10  task  clusters  and  for  the  aggregate.  The  overall  model  (F-test)  was  significant 
for  all  but  two  of  these  clusters.  (The  data  did  not  fit  the  model  well  for  Task  Clusters  7  and 
10.  This  may  be  due  to  the  small  number  of  observations  available  for  the  clusters.)  For 
those  clusters  with  significant  F-tests,  experience  was  a  significant  explanatory  variable.  Al  as 
measured  by  the  ASVAB  Electronics  score  was  statistically  significant  for  four  of  those  clusters. 
The  aggregate  model,  in  which  the  dependent  variable  is  calculated  from  a  weighted  average 
of  performance  times,  was  significant  in  the  overall  F-test  as  well  as  the  two  predictors.  The 
coefficient  of  determination  was  0.4424,  which,  given  the  sample  size  and  the  nature  of  the 
data,  indicates  a  reasonably  good  fit. 

Figures  1  through  9  graphically  represent  the  estimated  production  curves  for  8  of  the  10 
task  clusters  and  for  the  aggregated  clusters.  (Clusters  7  and  10  were  omitted  due  to  lack  of 
fit.  A  larger  sample  size  is  needed  to  evaluate  the  significance  of  these  models.)  The  curves 
are  shown  with  productive  capacity  on  the  vertical  axis  and  experience  (months  in  AFS)  on 
the  horizontal  axis.  The  three  curves  represent  the  function  at  Electronics  Al  scores  of  70,  80, 
and  90.  In  all  graphs  the  lowest  curve  represents  an  electronics  score  of  70  while  the  highest 
curve  represents  a  score  of  90.  The  effect  of  electronics  score  on  productive  capacity  decreases 
as  the  curves  get  closer  together.  For  example,  electronics  score  has  a  weaker  effect  in  Task 
Ciuster  1  than  in  Task  Cluster  4.  The  effect  of  experience  on  productive  capacity  increases 
as  the  curves  become  steeper.  For  example,  experience  has  a  stronger  effect  in  Task  Cluster 
1  than  in  Task  Cluster  4.  These  observations  may  occur  because  the  tasks  in  Cluster  1  are 
administrative  whereas  Cluster  4  tasks  involve  electronics  skills.  Administrative  tasks  are  not 
expected  to  be  as  sensitive  to  electronics  scores  as  are  tasks  directly  involving  electronics 
skills.  However,  administrative  tasks  may  be  more  sensitive  to  experience  than  are  electronics 
tasks. 

In  addition  to  the  logistic  model,  a  log-linear  model  was  tested  and  yielded  a  comparable 
fit.  The  log-linear  model  does  not,  however,  confine  predicted  values  to  the  interval  (0,1). 
Negative  productive  capacity  or  productive  capacity  greater  than  1  could  be  predicted.  This 
inadequacy  was  the  primary  reason  for  the  selection  of  the  logistic  model  over  the  log-linear 
model. 


Production  Isoquants 


Another  interpretation  of  the  productive  capacity  model  involves  the  concept  of  production 
isoquants.  That  is,  all  pairs  of  values  of  experience  and  aptitude  which  yield  a  fixed  level  of 
productive  capacity  can  be  determined.  By  solving  the  productive  capacity  equation  for  experience 
(or,  alternatively,  for  aptitude),  the  relationship  between  experience  and  aptitude  can  be  determined 
for  a  fixed  level  of  productive  capacity: 

Xi  =  [In  (P/(1-P))  -  bo  -  b2X2]/bi. 


Figure  10  graphs  this  function  for  selected  values  of  productive  capacity  using  the  estimated 
regression  parameters.  The  horizontal  line  at  48  months  indicates  the  end  of  the  first  term. 
From  the  isoquants  one  can  determine  the  rate  of  substitution  between  aptitude  and  experience 
to  yield  a  given  level  of  productive  capacity.  For  example,  the  average  airman  with  an  eiectronics 
score  (E-score)  of  90  and  IS  months  of  experience  can  perform  at  a  rate  equal  to  that  of  the 
average  airman  with  an  electronics  score  of  60  and  35  months  of  experience.  On  the  average, 
a  difference  of  about  1.6  points  in  E-score  is  equivalent  to  1  month  of  experience.  Thus,  one 
airman  who  scores  16  points  higher  than  another  airman  will  reach  a  given  level  of  productive 
capacity  an  average  of  10  months  sooner  than  the  airman  with  the  lower  electronics  score. 
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MONTHS 

Figure  1.  Estimated  Production  Curves  for  Task  Cluster  1 
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Figure  2.  Estimated  Production  Curves  for  Task  duster  2 
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Estimated  Production  Curves  for  Task  Cluster  3 


Figure  4.  Estimated  Production  Curves  for  Task  Cluster  4 
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Figure  5.  Estimated  Production  Curves  for  Task  Cluster  5 
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Figure  6.  Estimated  Production  Curves  for  Task  Clusters 
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Estimated  Production  Curves  for  Task  Cluster  8 


MONTHS 

Figure  8.  Estimated  Production  Curves  for  Task  Cluster  9 
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Figure  10.  Estimated  Production 


The  isoquant  graph  can  also  be  utilized  to  estimate  the  expected  productive  capacity  of  an 
individual  with  a  given  electronics  score  at  a  given  point  in  his  or  her  career.  For  example, 
at  the  end  of  the  first  term  (48  months)  an  airman  with  an  electronics  score  of  40  would  reach 
an  average  productive  capacity  of  0.5  (i.e.,  50%  of  the  maximum  productive  capacity  estimated 
for  the  328X0  work  force).  An  individual  with  an  electronics  score  of  79  would  reach  an 
average  productive  capacity  of  0.65  at  the  end  of  the  first  term.  If  the  first-term  productivity 
objectives  are  defined,  the  isoquant  graph  can  be  used  to  select  minimum  enlistment  standards. 
For  example,  if  it  is  desired  that  all  airmen  reach  a  productive  capacity  of  0.65  by  the  end  of 
the  first  term,  only  airmen  scoring  79  and  higher  should  be  admitted  into  the  AFS.  Airmen 
with  electronics  scores  less  than  79  are  not  expected  to  reach  a  productive  capacity  of  0.65 
until  after  the  end  of  the  first  term. 

The  productive  capacity  model  is  not  only  an  essential  component  of  the  TTP  integrated 
model,  but  also  provides  a  useful  method  for  examining  the  relationships  among  experience, 
aptitude,  and  productivity.  As  illustrated  above,  isoquants  can  be  used  to  estimate  the  rate  of 
substitution  between  experience  and  aptitude  to  yield  a  constant  level  of  productivity.  The 
model  can  be  used  to  predict  the  productive  capacity  of  an  individual  with  a  given  level  of 
aptitude  at  a  given  point  in  his  or  her  career.  Airmen  can  then  be  selected  based  on  aptitude 
to  ensure  that  they  will  meet  a  required  level  of  productivity  by  a  given  point  in  their  career. 


Attrition  Model 


Attrition  can  be  modeled  as  the  probability  of  an  individual  remaining  in  service  after  a 
given  number  of  months.  This  probability  may  be  conditional  upon  the  mental  aptitude  of  the 
individual.  Data  were  obtained  from  the  Defense  Manpower  Data  Center  (DMDC)  to  estimate 
the  attrition  model.  Members  of  the  1982  cohort  group  were  tracked  over  48  months  to 
determine  their  time  of  separation.  The  data  indude  Air  Force  specialty  (AFS),  Electronics 
Score,  and  months  of  service  at  separation  for  the  cohort  group  under  study.  To  model  attrition 
for  the  AFS  328X0,  a  subset  of  the  DMDC  data  was  taken.  This  subset  included  all  individuals 
in  any  AFS  beginning  with  32.  There  were  not  enough  data  points  to  build  a  model  using 
only  AFS  prefix  328.  The  authors  believe  that  extending  the  scope  of  the  model  to  all  AFS 
32's  will  improve  the  model's  confidence  and  still  represent  the  specialty  under  study. 

The  airmen  in  AFS  32  were  stratified  into  groups  defined  by  Electronics  Score  (60-69,  70-79, 
80-89,  90-99).  For  each  group,  the  number  and  proportion  of  airmen  remaining  at  the  end  of 
each  month  during  the  first  term  were  accumulated.  Figure  11  graphs  the  percent  remaining 
in  each  of  the  four  aptitude  groups  over  the  first  48  months  of  service.  The  graph  shows  that 
in  the  early  months,  the  loss  rate  is  higher  for  the  lower  aptitude  groups.  The  loss  rates 
converge  at  the  end  of  the  term  and  are  virtually  the  same  across  aptitude  groups  in  the  48th 
month.  The  loss  in  the  last  month  represents  airmen  who  chose  not  to  re-enlist. 


A  mathematical  model  was  developed  to  describe  the  probability  of  remaining  in  service  as 
a  function  of  aptitude  and  time.  This  model  has  the  general  form: 

r(x,t)  *  bo  +  bi  In  [(t  +  s(x))  /  (48  -  t)]  +  b4X 

s(x)  =  exp(b2  +  bax) 
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where  r(x,t)  is  the  probability  of  an  airman  within  an  aptitude  index  of  x  remaining  in 
service  after  t  months  (t  =  1  to  48) 

X  is  the  aptitude  index  (electronics  score) 

t  is  the  number  of  months  in  service 

bo.  bi,  b2,  bs.  b4  are  parameters  to  be  estimated 

The  above  model  can  be  described  as  a  logistic  function  (in  terms  of  t)  with  a  phase  shift. 
This  phase  shift,  which  depends  on  aptitude,  is  represented  by  s(x).  The  modeled  phase  shift 
increases  exponentially  as  aptitude  (measured  by  electronics  score)  increases  linearly.  Since 
the  model  is  not  implicitly  linear,  a  gradient  search  technique  was  used  together  with  multiple 
regression  to  estimate  the  parameters: 


Parameter 

Estimate 

bo 

66.0881 

bi 

-5.5569 

b2 

-3.8500 

b3 

0.0654 

b4 

0.2679 

As  indicated  by  the  R^,  a  remarkably  good  fit  was  obtained.  Figure  12  graphs  the  predicted 
attrition  for  the  four  aptitude  groups,  along  with  the  extrapolated  curve  for  the  50-59  aptitude 
group. 

The  percent  remaining  has  a  steeper  decline  early  and  late  in  the  term.  During  training, 
the  decline  is  steeper  for  lower  aptitude  levels.  The  percent  remaining  in  the  48th  month  was 
not  predicted  since  expected  productivity  and  cost  for  any  given  month  are  affected  by  the 
force  remaining  at  the  end  of  preceding  month. 


Cost  Model 


Cost  can  also  be  modeled  as  a  function  of  time  in  service  and  mental  aptitude.  Military 
pay  increases  as  time  in  service  increases,  due  to  both  promotions  and  longevity  increases. 
The  rate  at  which  salaries  increase  may  also  depend  upon  mental  aptitude  (e.g.,  more  rapid 
promotion  may  be  linked  to  higher  aptitude).  Training  costs  occur  during  the  first  10  months. 
These  training  costs  may  be  extended  over  a  longer  period  of  time  if  some  individuals  must 
repeat  training  courses.  Recruiting  costs  occur  just  before  the  term  begins.  This  initial  cost 
may  vary  by  mental  aptitude. 

Information  on  costs  broken  down  by  aptitude  could  not  be  obtained.  Recruiting  Service 
had  no  recruiting  cost  information  identified  by  mental  categories  or  aptitude  levels.  Similarly, 
Air  Training  Command  did  not  have  information  on  training  costs  broken  down  by  aptitude. 
Although  "ATC  Cost  Factors"  (June  1987)  showed  average  training  costs  broken  down  by  AFS, 
the  information  was  useful,  but  did  not  provide  the  level  of  distinction  necessary  to  adequately 
model  cost  as  a  function  of  aptitude.  The  cost  model  is  consequently  constant  across  aptitude 
groups.  The  cost  model  includes  initial  recruiting  costs  and  training  costs  in  the  first  10  months. 
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Predicted  Attrition  by  Electronics  Score 


MBitary  pay  is  included  throughout  the  first  term,  with  increases  following  an  average  promotion 
schedule.  Figure  13  graphs  the  cost  model  derived  from  the  cost  data  extracted  from  "Air 
Training  Command  Cost  Factors"  (June  1987). 


Distribution  of  Aptitude 

For  reasons  discussed  under  Model  Integration,  the  distribution  of  aptitudes  among  airmen 
must  be  estimated.  The  data  obtained  from  DMDC  for  the  attrition  model  were  used  to  estimate 
this  distribution.  Figure  14  graphs  the  relative  frequency  for  each  of  six  aptitude  groups. 

To  see  the  effects  of  excluding  airmen  who  score  below  a  given  value  on  the  electronics 
component  of  the  ASVAB,  a  truncated  distribution  can  be  computed.  If  f(x)  is  the  relative 
frequency  associated  with  an  electronics  score  of  x,  then  fm(x)  -  f(x)/(1-  I  f(i))  is  the  relative 
frequency  associated  with  that  electronics  score  within  the  subpopulation  defined  by  excluding 
those  with  E-scores  less  than  m.  Figure  15  displays  the  relative  frequencies  with  the  full 
population  for  each  of  six  aptitude  groups,  together  with  the  relative  frequencies  obtained  by 
truncating  the  population  at  various  minimum  electronics  scores.  These  relative  frequencies  will 
be  used  to  estimate  average  cost  per  productive  unit  for  various  minimum  electronics  score 
cutoffs. 


V.  MODEL  INTEGRATION  AND  SOLUTION 


Integration 

Once  estimated,  the  models  must  be  integrated  into  a  meaningful  objective  function.  To 
develop  this  function,  it  is  helpful  to  consider  two  intermediate  functions:  expected  productive 
capacity  and  expected  cost.  Expected  first-term  productive  capacity  for  an  individual  with  a 
given  E-score  (x)  can  be  described  by  the  following  equation: 


48 

P(x)  =  1 

'  t-o 

where  P(x) 

X 


t 

r(x,t) 


P(x,t) 


r(x,t)  p(x,t) 

=  Expected  first-term  productive  capacity  for  an  individual  with  E-score  of  x 
=  E-score 

=  Time  in  service  (months) 

»  Probability  that  an  individual  with  E-score  of  x  is  still  in  service  after 
t  months 

=  Productive  capacity  for  an  individual  with  an  E-score  of  x  and  t  months 
of  experience. 
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ELECTRONICS  SCORE  CUT  OFFS 


Figure  16  graphs  the  expected  productive  capacity  for  the  first  term  for  an  average  airman 
in  each  of  five  aptitude  groups.  The  productive  capacity  is  zero  for  all  aptitude  levels  until 
the  eleventh  month.  The  height  of  the  curve  increases  as  aptitude  increases.  The  curves  for 
any  two  levels  of  aptitude  do  not  intersect  in  the  first  term.  This  implies  that  the  cumulative 
expected  productive  capacity  will  increase  as  level  of  aptitude  increases.  For  any  aptitude 
level,  the  expected  productive  capacity  function  increases  until  about  the  middle  of  the  final 
year  of  the  first  term.  At  that  point,  it  reaches  a  maximum  and  decreases  until  the  end  of  the 
term.  Up  until  the  maximum  point,  productive  capacity  increases  at  a  rate  greater  than  that 
of  attrition.  Beyond  that  point,  the  attrition  rate  increases  faster  than  productive  capacity.  That 
is.  productivity  does  not  increase  fast  enough  to  compensate  for  attrition  late  in  the  term.  This 
may  be  caused  by  "early  outs"  which  some  airmen  choose  to  take  at  this  time. 

Figure  17  presents  a  bar  chart  representing  the  total  expected  productive  capacity  for  the 
ftrst  term  for  five  aptitude  groups.  The  total  expected  productive  capacity  ranges  from  12.62 
for  the  50-59  group  to  19.43  for  the  90-99  group.  In  the  first  term,  the  average  airman  with 
an  E-score  between  50  and  59  produces  an  amount  equivalent  to  a  worker  performing  at  the 
maximum  achievable  level  for  12.62  months.  Similarly  the  average  airman  in  the  90-99  group 
produces  the  equivalent  to  19.43  months.  This  implies  that  over  the  first  term,  the  average 
airman  in  the  90-99  E-score  group  is  over  50%  more  productive  than  the  average  airman  in 
the  50-59  group. 


Expected  first-term  cost  for  an  individual  with  a  given  E-score  can  be  described  by; 
48 

C(x)  =  2^  r(x.t)  c(x.t) 


where  C(x) 

X 


t 

r(x.t) 


c(x.t) 


-  Expected  first-term  productive  capacity  for  an  individual  with  E-score  of  x 
»  E-score 

=  Time  in  service  (months) 

=  Probability  that  an  individual  with  E-score  of  x  is  still  in  service  after  t 
months 

=  Cost  to  the  Air  Force  of  an  individual  with  E-score  x  in  month  t. 


Figure  18  graphs  the  expected  monthly  cost  over  the  first  term  (beginning  with  the  third 
month)  for  each  of  five  aptitude  levels.  Since  the  cost  model  does  not  vary  by  aptitude, 
differences  in  expected  cost  among  aptitude  levels  are  due  to  differences  in  attrition  rates.  At 

month  zero,  the  recruiting  cost  is  the  same  for  all  aptitude  levels.  Attrition  has  not  yet  taken 

effect.  The  first  month  includes  the  cost  of  basic  training.  Attrition  still  has  no  effect.  The 
second  month’s  costs  include  basic  training  during  the  first  half  and  technical  training  during 
the  second  half.  Attrition  has  a  small  effect  in  the  second  month.  Costs  in  months  3  through 

TO  include  technical  training.  Expected  costs  are  lower  for  lower  aptitude  levels  because  of 

their  higher  attrition  rates.  After  training,  expected  cost  for  any  fixed  aptitude  level  decreases 
until  the  point  of  pay  increase.  It  then  jumps  to  compensate  for  the  pay  increase  and  declines 
until  the  next  pay  increase,  and  then  the  process  is  repeated. 
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Figure  17.  Expected  First-Term  Productive  Capacity 
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Figure  18.  Expected  Cost  per  Month  tor  an  Entry  Cohort 


Figure  19  presents  a  bar  chart  representing  the  total  expected  first-term  cost  for  the  five 
aptitude  groups.  The  total  expected  cost  ranges  from  $58,239.00  for  the  50-59  E-score  group 
to  $63,436.00  for  the  90-99  E-score  group.  Total  expected  cost  is  higher  for  higher  aptitude 
groups  because  airmen  with  higher  aptitude  are  more  likeiy  to  remain  in  service  than  airmen 
with  lower  aptitude. 

The  ratio  C(x)/P(x)  represents  the  expected  cost  per  productive  unit  over  the  first  term.  By 
minimizing  this  ratio  with  respect  to  x  (E-score),  one  can  obtain  the  optimum  E-score  to  minimize 
cost  per  productive  unit.  Figure  20  presents  a  bar  chart  representing  the  expected  cost  per 
productive  unit  for  each  of  five  aptitude  groups.  The  expected  cost  per  productive  unit  ranges 
from  $3,264.00  for  the  highest  aptitude  group  to  $4,614.00  for  the  lowest  group  displayed. 
Thus,  to  produce  the  same  amount  of  work  in  the  first  term,  it  would  cost  over  40%  more  to 
employ  airmen  with  E-scores  between  50  and  59  than  airmen  with  E-scores  between  90  and 
99. 


The  solution  of  this  minimization  problem  would  imply  that  all  new  recruits  should  be  selected 
with  an  E-score  equal  to  the  optimum  value  of  x.  It  would  be  impractical  to  impose  such  a 
constraint  on  the  selection  of  new  AFS  members.  The  distribution  of  the  pool  from  which  new 
recruits  are  selected  must  be  considered.  If  f(x)  is  the  probability  density  function  of  E-scores 
for  the  population  of  potential  recruits,  then  fm(x)  =  f(x)\1-^2  f(i))  is  the  conditional  probability 
density  function  for  the  population  of  potential  recruits  with  an  1-score  of  at  least  m.  Productive 
capacity  and  cost  can  then  be  modeled  as  functions  of  a  minimum  allowable  E-score. 

EIP(m)l  =  1  fm(x)  P(x) 

x>  ■m 

EIC(m)l  =  2  fm(x)  C{x) 

x>  >m 

Where:  m  is  the  minimum  allowable  E-score 

E[P(m)]  is  the  expected  (average)  first-term  productive  capacity  for  the  subpopulation 
of  potential  recruits  with  E-scores  of  at  least  m 

E[C(m)]  is  the  expected  (average)  first-term  cost  for  that  subpopulation 

fm(x)  is  the  conditional  probability  density  function  of  E-scores  for  the  subpopulation 
of  potential  recruits  with  E-scores  of  at  least  m 

P(x)  and  C(x)  are  as  defined  earlier. 


By  minimizing  the  ratio  E[C(m)]/E[P(m)]  with  respect  to  m,  one  can  find  the  optimum 
minimum  E-score  standard  for  a  given  distribution  of  E-scores  among  a  recruiting  pool.  The 
solution  is  given  in  the  next  section. 


Optimal  Solution 

The  optimal  solution  is  obtained  by  comparing  the  computed  expected  cost  per  productive 
unit  among  the  subpopulations  obtained  by  imposing  different  E-score  cutoffs. 
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Figure  19.  Expected  First-Term  Cost  by  Electronics  Score 
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Figure  20.  Expected  First-Term  Cost  per  Productive  Unit 


Figure  21  presents  a  bar  chart  representing  the  average  first-term  cost  per  productive  unit 
for  the  subpopulations  obtained  by  imposing  minimum  electronics  score  requirements  of  50,  60, 
70,  80  and  90.  The  average  cost  per  productive  unit  ranges  from  $3,264.00  when  a  minimum 
electronics  score  of  90  is  imposed  to  $3,968.00  when  the  cutoff  is  50.  Thus,  the  lowest  average 
cost  per  productive  unit  !s  obtained  when  the  minimum  required  electronics  score  for  admission 
to  the  AFS  is  90.  Under  a  different  cost  model  the  solution  might  change. 

Sensitivitv  Analysis 

Since  the  cost  model  does  not  depend  on  aptitude,  the  optimum  aptitude  group  is  the 
group  with  the  highest  expected  productive  capacity.  Sensitivity  analysis  allows  an  investigation 
as  to  how  this  solution  might  change  if  a  different  cost  model  was  appropriate.  Of  particular 
interest  is  the  limit  to  which  the  costs  associated  with  the  optimum  aptitude  group  can  be 
increased  without  changing  the  current  solution. 

If  the  total  expected  cost  for  the  90-99  group  were  increased  by  $5,035.00  (roughly  8%), 
the  average  cost  per  productive  unit  for  the  subpopulation  defined  by  truncating  at  an  electronics 
score  of  80  would  be  approximately  the  same  as  the  cost  per  productive  unit  for  the  subpopulation 
truncated  at  90  (i.e.,  the  difference  in  productive  capacity  between  the  two  cutoffs  would  not 
compensate  for  the  difference  in  cost).  Any  increase  beyond  this  would  select  members  with 
less  than  the  highest  category  of  aptitude  scores.  Such  a  difference  is  not  implausible,  which 
suggests  that  a  more  accurate  cost  model  should  be  developed  to  reduce  the  uncertainty  of 
the  cutoff  decision. 

Increased  training  costs  for  lower  aptitude  groups  would  evoke  a  stronger  argument  for 
selecting  from  the  highest  aptitude  groups.  Increased  recruiting  costs  and  military  pay  for 
higher  aptitude  groups  would  influence  the  solution  in  the  direction  of  choosing  lower  aptitude 
individuals.  Other  modifications  to  the  cost  model  would  yield  varied  results,  thus  stressing 
the  need  for  greater  accuracy  and  reliability  in  the  cost  model. 


VI.  CONCLUSIONS  AND  RECOMMENDATIONS 
Conclusions 


This  work  has  demonstrated  how  a  model  can  be  formulated  and  solved  to  arrive  at  an 
optimum  aptitude  standard  (measured  by  electronics  score)  which  minimizes  cost  per  unit  of 
production.  The  objective  model  consists  of  four  components: 

1.  Productive  Capacity  Model 

2.  Attrition  Model 

3.  Cost  Model 

4.  Distribution  of  Aptitudes 

The  objective  model,  together  with  its  components,  yields  significant  findings: 

1.  Higher  quality  people  achieve  a  specified  level  of  productivity  more  rapidly;  e.g.,  the 
average  airman  with  an  electronics  score  of  80  and  18  months  of  experience  is  estimated  to 
perform  at  a  rate  equal  to  that  of  the  average  airman  with  an  electronics  score  of  50  and  36 
months  of  experience. 
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Figure  21.  Average  Cost  per  Productive  Unit  for  Electronics  Score  Cut  Offs 


2.  The  quaiity/experience  trade-off  is  approximately  16  points  per  10  months;  i.e.,  if  one 
airman  scores  16  points  higher  than  another,  he  will  reach  a  given  level  of  productive  capacity 
an  estimated  10  months  sooner  than  the  airman  with  the  lower  electronics  score. 

3.  The  attrition  model  demonstrates  that  the  loss  rate  during  training  is  higher  for  the  iower 
aptitude  groups. 

4.  Adjusting  for  attrition,  it  is  estimated  that  the  average  airman  in  the  90-99  electronics 
score  group  is  over  50%  more  productive  than  the  average  airman  In  the  50-59  group  during 
the  first  term. 

5.  Using  a  simplified  cost  model  which  assumes  no  differences  in  recruiting,  training  or 
sustainment  costs  over  the  first  term  as  a  function  of  quality,  high  quality  people  are  more 
cost  effective  over  the  first  term.  For  example,  it  costs  an  estimated  40%  more  for  a  group 
of  airmen  with  electronics  scores  between  50  and  59  than  a  group  of  airmen  with  electronics 
scores  between  90  and  99  to  produce  the  same  amount  of  work  in  the  first  term. 

The  results  of  this  research  must  be  caveated  by  the  fact  that  the  study  is  limited  to 
production-oriented  tasks  in  only  one  Air  Force  specialty.  These  results  would  not  be  expected 
to  hold  true  either  in  the  same  direction  or  degree  for  all  specialties.  However,  it  is  reasonable 
to  believe  that  these  results  are  typical  for  the  majority  of  specialties  requiring  a  high  degree 
of  skill  and  aptitude  and  dominated  by  production-oriented  tasks  over  the  first  term. 

It  is  also  cautioned  that  these  results  apply  only  to  the  first-term  "life  cycle."  If  the  selection 
model  were  to  consider  the  loss  rates  after  the  first  term  and  the  resulting  production  over  the 
second  and  subsequent  terms,  the  results  might  be  different.  In  addition,  the  development  of 
and  requirement  for  leadership  and  administrative  skills  beyond  the  first  term  would  create 
another  dimension  to  investigate  when  considering  the  cost-efficiency  of  skilled  personnel. 

Nevertheless,  with  these  caveats  in  mind,  it  is  fair  to  conclude  that  this  research  indicates 
that  as  long  as  recruiting  and  sustainment  costs  are  not  significantly  higher  for  higher  quality 
personnel,  the  Air  Force  and  the  taxpayers  are  best  served  by  setting  their  quality  standards 
as  high  as  possible  for  AFSs  that  are  directly  similar  in  work  and  requirements  to  Avionic 
Communications  Specialists,  328X0. 


Recommendations  for  Further  Research 


Additional  research  is  required  to  replicate  Phases  1  and  3  for  additional  AFSs  in  order  to 
test  the  method’s  broader  applicability  and  confirm  its  practical,  economic,  and  operational 
potentials.  Naturally,  one  should  expect  that  different  specialties  will  bring  different  analytic 
complexities,  and  several  applications  will  have  to  be  completed  before  the  approach's  generality 
and  cost-effectiveness  can  be  judged  as  satisfactory. 

Further  applications  of  the  approach  would  also  offer  opportunities  to  develop  different  and 
perhaps  more  efficient  protocols  for  collecting  objective  data  of  the  types  needed  to  validate 
the  subjective  data,  to  select  tasks  for  direct  observation  that  could  improve  the  statistical 
properties  of  estimates  and  hypothesis  tests,  and  to  devise  improved  formats  and  mechanisms 
for  collecting  subjective  assessments  from  supervisors. 
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In  order  to  fully  evaluate  the  TTP  measurement  approach  and  adequacy  of  the  related 
models,  data  are  needed  for  more  individuals  in  more  Air  Force  specialties.  These  data  include 
the  information  collected  on  the  TTP  Questionnaire,  as  well  as  sufficient  information  on  attrition 
and  costs  broken  down  by  experience  and  mental  quality  and  other  attributes  that  might  augment 
the  model.  Larger  sample  sizes  will  allow  the  testing  of  more  complex  regression  modeis  while 
preserving  sufficient  degrees  of  freedom  to  evaluate  the  significance  of  the  models.  Repeating 
the  process  for  several  Air  Force  specialties  will  help  to  verify  the  validity  of  the  TTP  process 
and  the  resulting  models. 

A  more  accurate  cost  model  is  needed  to  replace  or  confirm  the  solution  found  here. 
Recruiting  and  training  costs  must  be  broken  down  by  aptitude.  Promotion  schedules  must 
similarly  be  disaggregated.  If  such  data  are  not  available  in  aggregate  form,  the  itemized  costs 
per  individual  could  be  merged  with  demographic  data  (including  aptitude  measures)  from  DMDC 
and  aggregated  into  a  usable  form.  This  assumes  that  records  are  kept  on  each  individual 
airman. 

The  TTP  model  should  be  extended  beyond  a  single  Air  Force  specialty.  The  models 
developed  here  should  be  validated  using  two  or  three  additionai  AFSs,  obtaining  an  optimai 
solution  for  each  independently.  Then  all  specialties  within  a  certain  classification  (e.g..  Electronics 
Al)  could  be  analyzed  together,  assuming  a  limited  number  of  recruits.  A  mathematical 
programming  algorithm  could  be  used  to  optimally  allocate  recruits  to  the  specialties  based  on 
electronics  scores  and  availability  and  practicality  constraints.  Finally,  all  Air  Force  specialties 
could  be  considered.  A  more  complex  algorithm  could  be  used  to  optimally  allocate  the  recruits 
to  the  specialties  based  on  the  vector  of  Mechanical,  Administrative,  General,  and  Electronics 
Al  scores. 
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APPENDIX  A:  FINAL  TASK  CLUSTERS 


Task 
Group  # 

Task  difficulty 

Mean  SO 

#  Of 

Tasks 

#  of 

WTPT  tasks 

1 

4.047 

0.683 

5 

1 

2 

3.655 

0.370 

13 

4 

3 

5.997 

0.759 

9 

5 

4 

4.496 

0.370 

10 

2 

5 

5.642 

0.677 

11 

6 

6 

5.412 

0.817 

7 

1 

7 

5.349 

0.707 

5 

1 

8 

5.977 

0.928 

11 

3 

9 

4.273 

0.707 

5 

0 

10 

5.487 

0.656 

11 

2 
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APPENDIX  A  (Continued) 


DEFINITION  OF  TTP  TASK  CLUSTERS 


Task  Cluster  1  -  Maintenance  Administration  and  inspection 

Locate  Part  or  Stock  Numbers  in  Technical  Publications  (Form  781) 

*  Make  Entries  on  Maintenance  Data  Collection  Record  Forms  (AFTO  Forms  349) 
Make  Entries  on  Reparabie  item  Processing  Tag  Forms 

Inspect  Avionic  Equipment  for  Corrosion  and  Perform  Corrosion  Controi 
Locate  Maintenance  Information  in  Technical  Publications 

Task  Cluster  2  •  Flightline  Maintenance 

*  Safety-Wire  or  Bond  System  Components 
Test  Continuity  of  Coaxial  Cables 
Preset  Frequencies  in  UHF  Control  Units 
Remove  or  Replace  UHF  Control  Units 

*  Remove  or  Replace  UHF  Receiver-Transmitters 
Operationally  Check  Interphone  Systems 
Remove  or  Replace  Interphone  Cord  Components 
Remove  or  Replace  Interphone  Cords 

*  Remove  or  Replace  Interphone  Station  Control  Units 
Remove  or  Replace  Interphone  Monitor  Control  Units 
Remove  or  Replace  UHF  Antennas 

Operate  AGE,  such  as  Power  Units,  Heaters,  or  Light  Carts 

*  Inspect  Parts  Received  from  Supply  or  Manufacturer 

Task  Cluster  3  -  Flightline  Troubleshooting 

isolate  Malfunctions  in  Avionic  Systems  Wiring  or  Cables 

*  Repair  Avionic  System  Wiring  or  Cables 

*  Remove  or  Replace  Radio  Frequency  (RF)  Coaxial  Connectors 
Solder  Avionic  System  Wiring 

*  Isoiate  Malfunctions  in  UHF  Systems 

*  T race  Circuits  or  Signals  Using  Wiring  Diagrams  or  Schematics 
Isoiate  Maif unctions  in  Interphone  Systems 
Operationally  Check  Systems  Using  Flightline  Test  Equipment 

*  Remove  or  Repiace  Muitipie  Wire  Plugs 
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Appendix  A  (Continued ) 


Task  Cluster  4  -  Shop  Maintenance 

Remove  or  Replace  Receiver-Transmitter  Subassemblies 
Remove  or  Replace  UHF  Control  Unit  Subassemblies 
Bench  Check  Interphone  Monitor  Control  Units 
Bench  Check  Interphone  Station  Control  Units 
Remove  or  Replace  interphone  Control  Subassemblies 
Remove  or  Replace  Interphone  Control  Components 

*  Isolate  Malfunctions  in  Interphone  Cords 

*  Isolate  Malfunctions  in  Interphone  Monitor  Control  Units 
Isolate  Malfunctions  in  Interphone  Station  Control  Units 
Bench  Check  UHF  Remote  Channel  Indicators 

Task  Cluster  5  -  UHF  Shop  Maintenance 

Diagnose  Mockup  Malfunctions 
Align  UHF  Control  Units 
Align  UHF  Receiver-Transmitters 

•  Bench  Check  UHF  Receiver-Transmitters 

*  Isolate  Malfunctions  in  UHF  Receiver-Transmitters 

•  Bench  Check  UHF  Control  Units 

Isolate  Malfunctions  in  UHF  Control  Units 
Remove  or  Replace  UHF  Control  Unit  Components 

•  Remove  or  Replace  UHF  Receiver-T ransmitter  Components 

•  Set  Up  UHF  System-Peculiar  Test  Equipment 

•  Perform  Time  Compliance  Technical  Order  (TCTO)  Modifications  of  Avionic  Systems 

Task  Cluster  6  -  Flightline  HF  Maintenance 

Isolate  Malfunctions  in  HF  Systems 
Remove  or  Replace  HF  Control  Units 

Remove  or  Replace  HF  Couplers 

*  Remove  or  Replace  HF  Receiver-Transmitters 
Operate  Associated  Systems  Checking  HF  Systems 
Operationally  Check  HF  Systems  Using  FTE 
Remove  or  Replace  HF  Antennas 

Task  Cluster  7  -  HF  Control  Box  In-Shop  Maintenance 

Align  HF  Control  Boxes 
Isolate  Malfunctions  in  HF  Control  Boxes 
Remove  or  Replace  HF  Control  Box  Components 

Remove  or  Replace  HF  Control  Box  Subassemblies 

*  Bench  Check  HF  Control  Boxes 
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Appendix  A  (Concluded) 

Task  Cluster  8  •  Receiver-Transmitter/HF  Coupler  Shop  Maintenance 

Remove  or  Replace  HF  Coupler  Components 
Remove  or  Replace  HF  Coupler  Subassemblies 
Remove  or  Replace  HF  Receiver-Transmitter  Components 

*  Remove  or  Replace  HF  Recelver-T ransmltter  Subassemblies 

*  S^  Up  HF  System-Peculiar  Test  Equipment 
Align  HF  Couplers 

Isolate  Malfunctions  in  HF  Couplers 
Bench  Check  HF  Couplers 
Align  HF  Receiver-Transmitters 
Bench  Check  HF  Receiver-Transmitters 

*  Isolate  Malfunctions  in  HF  Receiver-Transmitters 


Task  Cluster  9  -  Flightline  VHF  Maintenance 

Remove  or  Replace  Very  High  Frequency  (VHF)  Amplitude  Modulated  (AM)  Radio  Antenna 

Remove  or  Replace  VHF  Control  Units 

Remove  or  Replace  VHF  AM  Receiver-Transmitters 

Operate  Associated  Systems  Checking  VHF  AM  Systems 

Operationally  Check  VHF  AM  Systems  Using  Right  Test  Equipment 

Task  Cluster  10  •  Shop  VHF  Maintenance 

Align  VHF  AM  Control  Units 

Bench  Check  VHF  AM  Control  Units 

Remove  or  Replace  VHF  AM  Control  Unit  Components 

Remove  or  Replace  VHF  AM  Receiver-Transmitter  Subassemblies 

Remove  or  Replace  AM  Receiver-Transmitter  Components 

Isolate  Malfunctions  in  VHF  AM  Systems 

•  Align  VHF  AM  Receiver-Transmitters 

•  Bench  Check  VHF  AM  Receiver-Transmitters 

Isolate  Malfunctions  in  VHF  AM  Receiver-Transmitters 
Isolate  Malfunctions  in  VHF  AM  Control  Units 

Set  Up  VHF  AM-Pecullar  Test  Equipment _ 

‘Overlaps  the  Walk-Through  Performance  Test 
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APPENDIX  B; 


TIME  TO  PROFICIENCY  QUESTIONNAIRE 


TIME  TO  PROFICIENCY  HTP^  QUESTIONNAIRE 


Purpose  of  Questionnaire: 

The  purpose  of  this  study  is  to  collect  information  on  how  work  experience  affects  a 
worker's  efficiency.  For  the  purposes  of  this  study,  efficiency  is  defined  as  having  two 
parts;  the  amount  of  time  the  worker  requires  to  perform  the  task  satisfactorily,  and 
the  amount  of  time  needed  to  spend  supervising  the  worker.  This  questionnaire  asks 
for  estimates  of  both  performance  time  and  supervision  time  k}r  each  of  the  task 
areas  which  are  included  in  this  AFS.  This  information  is  confidential  and  for  study 
purposes  only,  and  will  not  be  used  for  formal  evaluations. 
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Inatructlona  For  Comoletlng  The  Questionnaire: 


Overview: 

In  order  to  complete  this  questionnaire  you  will  first  need  to  identify 
the  workers,  whom  you  supervise,  who  have  a  328X0  APS.  After  providing 
information  on  each  worker  (Step  One  below),  you  will  complete  five 
steps  (Steps  2  through  6  below),  for  each  of  ten  Task  Groups.  In 
these  steps  you  will  be  evaluating  the  performance  of  your  workers  in 
each  of  these  Task  Groups.  Each  Task  Group  represents  an  area  of 
responsibility  within  the  328X0  APS. 

The  steps  you  will  complete  are  outlined  briefly  below,  and  are  then 
described  in  detail.  An  example  of  a  completed  questionnaire  follows 
the  detailed  instructions. 

Step  One;  Complete  Worker  Identification  Information 

in  Step  One  you  will  complete  information  which  will  identify  the 
workers  under  your  supen/ision  who  perform  the  tasks  listed.  You 
will  complete  this  step  only  once. 

Step  Two:  Review  Task  Group  Description 

In  Step  Two  you  will  familiarize  yourself  with  the  various  tasks 
within  the  Task  Group  you  will  evaluate.  You  will  also  need  to  be 
familiar  with  performance  benchmarks  for  selected  tasks  for  the 
Task  Group.  You  will  repeat  this  step  for  each  Task  Group  rating. 

Step  Three:  Complete  Task  Group  Data  Column  One 

In  Step  Three  you  will  indicate  how  often  each  worker  performs  the 
tasks  in  the  Task  Group  you  are  evaluating.  You  will  repeat  this  step 
for  each  Task  Group  rating. 

Step  Four:  Complete  Task  Group  Data  Column  Two 

In  Step  Four  you  will  select  a  Benchmark  Worker  against  which  you  wil 
compare  the  other  workers.  You  will  repeat  this  step  for  each  Task 
Group  rating. 

Step  Rve:  Complete  Task  Group  Data  Column  Three 

In  Step  Rve  you  will  rate  each  worker  against  the  Benchmark  Worker 
you  selected  in  Step  Four,  in  Step  Rve  you  will  rate  the  workers  for 
performance  time.  You  will  repeat  this  step  for  each  Task  Group  rating. 

Step  Six:  Complete  Task  Group  Data  CQlumn_EQiif 

In  Step  Six  you  will  indicate  the  amount  of  time  you  supervise  the 
workers.  You  will  repeat  this  step  for  each  Task  Group  rating. 

Page  1 
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Detailed  Instructions: 

Step  One:  Complete  Worker  Identification  Information 

In  order  to  complete  Step  One  begin  by  turning  this  questionnaire  to 
page  26  (the  last  page).  You  will  notice  that  this  page  has  a  flap 
along  the  right  side  of  the  page.  Please  unfold  the  flap.  Notice  that 
this  page  asks  for  basic  information  on  each  of  the  workers  you 
supervise.  The  information  asked  for  on  this  page  is:  the  length  of 
time  (years  and  months)  that  each  worker  has  been  in  the  328X0  APS 
(not  including  Technical  School)  .each  worker's  Social  Security  Number 
(SSN),  and  each  worker's  name.  Once  you  have  completed  the  information 
on  this  page  turn  to  page  6.  When  you  turn  to  page  6  LEAVE  THE  FLAP 
ALONGlHERiGHTSDEOFPAGE26EXTENDED.  Leavingtheflap 
extended  will  help  you  to  complete  the  performance  and  supervision 
information  later  in  the  questionnaire. 

Two:  Review  Task  Group  Description 

in  order  to  complete  Step  Two  please  turn  to  page  6.  On  this  page 
you  will  notice  information  at  the  top  of  the  page  which  identifies 
the  particular  Task  Group  for  which  you  will  rate  the  workers  identified 
in  Step  One.  Following  the  Task  Group  identification  there  is  a  list  of 
tasks  included  in  the  Task  Group.  To  the  right  of  the  tasks  are  performance 
benchmarks.  The  performance  benchmarks  are  typical  times  to  complete 
the  tasks.  You  will  use  the  performance  benchmarks  to  select  a  Benchmark 
Worker  in  Step  Four.  Please  familiarize  yourself  with  the  Information 
on  this  page  and  begin  thinking  about  the  performance  of  your  workers 
in  the  tasks  on  this  page. 


ThreeL^omplete.Task  Group  Data  Column  One 

Indicate  at  the  top  of  the  next  page  of  the  questionnaire  if  any  of  the 
tasks  you  have  just  read  are  performed  by  any  of  the  workers  you 
supervise.  If  you  indicate  "NO”  (none  of  ^e  workers  you  supervise 
perform  any  of  the  tasks  on  the  previous  page)  then  turn  the  page  and 
begin  Step  Two  for  the  next  Task  Group.  If  you  indicate  "YES"  (at  least 
one  of  the  workers  you  supervise  perfoms  at  least  one  of  the  tasks  on 
the  previous  page)  then  continue  on  with  Step  Three. 

Next  please  indicate  in  Column  One  how  often  each  of  the  workers  you 
supervise  performs  the  tasks  in  the  Task  Group  you  are  evaluating. 
Please  use  the  following  codes: 

"R"  -  the  worker  performs  the  tasks  REGULARLY, 

"O'*  -  the  worker  performs  the  tasks  OCCASIONALLY,  or 
'*N'*  •  the  worker  NEVER  performs  the  tasks. 

Page  2 
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step  Four;  Complete  Task  Group  Data  Column  Two 

in  order  to  complete  Step  Four  you  will  need  to  select  a  Benchmark 
Worker  against  which  you  will  compare  ail  of  the  other  workers  for 
the  tasks  in  this  Task  Group.  Please  select  the  worker  who  generally 
performs  closest  to  the  performance  benchmarks  listed  on  the  previous 
page.  The  worker  you  select  should  perform  closest  to  the  times 
listed,  while  completing  the  work  to  an  acceptable  quality  level.  Please 
indicate  which  worker  you  select  by  checking  one  worker  in  Column  Two. 
Please  note  that  the  worker  you  select  as  the  Benchmark  Worker 
pertains  to  only  the  Task  Group  you  are  evaluating.  You  may  select 
that  worker  as  the  Benchmark  Worker  for  another  Task  Group,  but  you 
do  not  necessarily  have  to  do  so. 

Step  F1ve:Complete  Task  Group  Data  Column  Three 

You  will  complete  this  step  for  each  worker .  To  do  this  you  will 
compare  each  of  these  workers'  performance,  on  average,  to  that  of 
the  Benchmark  Worker  you  selected  in  Step  Four. 

Recall  the  amount  of  work  (in  this  Task  Group)  that  the  Benchmark 
Worker  can  complete  correct  in  one  hour.  Now  indicate  in  Column 
Three  the  amount  of  time  each  of  the  workers  requires  to  complete 
the  same  amount  and  quality  of  work.  The  amount  of  time  you  indicate 
in  Column  Three  can  be  greater  than,  less  than,  or  equal  to  one  hour.  For 
example,  the  work  that  the  Benchmark  Worker  completes  in  one  hour 
might  take  Worker  1  one  hour  and  15  minutes  to  complete  ,  while 
Worker  2  can  complete  the  work  in  forty  minutes. 

Step  Six:  Complete  Task  Group  Data  Column  Four 

in  order  to  complete  Step  Six  please  indicate  in  Column  Four  how  much 
time  you  would  require  to  supervise  each  worker  while  he  or  she  completes 
the  work  you  evaluated  in  Column  Three.  Please  note  that  the  times 
you  indicate  in  Columns  Three  and  Four  are  independent  A  worker  who 
requires  more  time  to  perform  the  work  may  or  may  not  require  more 
supenrision  time.  Supervision  time  can  include  quality  checks  of  the 
completed  work  as  well  as  direct  ”over  the  shoulder”  supervision  while 
the  work  is  being  performed.  Supervision  time  also  includes  any 
delegated  supervision. 


AFTH^YCXJ  HAVE  COMPLETED  STEP  SK  FOR  TVEFIRSTTASK  GROUP  PLEASE 

TURNTHEPASEANDCOMPLEIESTBJSTWOTHROUGHSKPORTHE 

RBWVNNGTASKGROUPS. 


THA^KYOU^/5T^'^AXHFO^YaJRCXX>=BVV^^ 
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Examplai 


As  you  can  sea  below  this  is  an  example  of  the  Worker  identification 
Information  page  that  has  been  completed  correctly. 
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After  reviewing  the  Task  Group  Description  Information,  the  supervisor 
has  completed  the  Task  Group  Data.  The  supervisor  has:  compieM  Column 
One  for  each  worker,  selected  a  Benchmark  Worker  in  Column  Two, 
compared  the  performance  of  the  other  workers  to  that  of  the  Benchmark 
Worker  in  Column  Three,  and  indicated  the  required  supervision  time 
for  each  worker  in  Column  Four, 
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(THIS  PAGE  WAS  IMTEI^ONALLY  LEFT  BLANK] 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _  (Please  complete  the  table  below.) 

NO  _ (Please  turn  to  the  next  Task  Area.) 


OOLUMMChE  OCXJUMviTWO  CCUJMN1VFE  CCLUMNRXR 


Worker 

How  Often 
Performs  Jobs 
In  This  Group 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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Renxwe  or  Replace  UHF  Antennas  26  Mins 

Operate  AGE.  Such  as  Power  Units,  Heaters,  or  Light  Carts  1 1  Mins. 

Inspect  Parts  Received  from  Supply  or  Manufacturer  5  Mins. 


TASK  AREA  DATA 

( 


Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 

YES _  (Please  complete  the  table  below.) 

NO  _  (Please  turn  to  the  next  Task  Area.) 


OOLUM^O^  OGLJLMMTWD  OOJUMNTPFS  OOLUMNRDUR 


Worker 

How  Often 
Performs  Jobs 
in  This  Group 
(R/O/N) 
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(Check  One) 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 

Time 

Isolate  Malfunctions  in  Avionic  Systems  Wiring  or  Cabies  1  Hr.  23  Mins. 
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Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _  (Piease  complete  the  table  below.) 

NO  _  (Please  turn  to  the  next  Task  Area.) 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 

Time 

Remove  or  Replace  Receiver-Transmitter  Subassemblies  1 8  Mins. 
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Isolate  Malfunctions  In  Interphone  Monitor  Control  Units  22  Mins. 

Isolate  Malfunctions  In  Interphone  Station  Control  Units  22  Mins. 

Bench  Check  UHF  Remote  Channel  Indicators  1 4  Mins. 


Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 

YES _  (Please  complete  the  table  below.) 

NO  _ (Please  turn  to  the  next  Task  Area.) 


OOULMJCNE  OCmmiV^O  CaUMNT-FE  OCUJwWFajR 


Worker 

How  Often 
Performs  Jobs 
In  This  Group 
(R'O/N) 
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Remove  or  Replace  UHF  Control  Unit  Components  27  Mins. 

Remove  or  Replace  UHF  Receiver-Transmitter  Components  1 5  Mins. 

Set-Up  UHF  System  Peculiar  Test  Equipment  1 1  Mins. 

Perform  Time  Compliance  Technical  Order  (TCTO)  Modifications  on  Avionic  Systems  1  Hr.  1 5  Mins. 


TASK  AREA  DATA 

Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 

YES _  (Please  complete  the  table  below.) 

NO  _  (Please  turn  to  the  next  Task  Area.) 


OOLJJM^ae 


How  Often 
Performs  Jobs 
In  This  Group 
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This  Task  Group  includes  the  foilowing  tasks  and  performance  benchmarks: 
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Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _  (Please  complete  the  table  below.) 

NO  _ (Please  turn  to  the  next  Task  Area.) 


OQUJMvJChE  OCUMnJTWO  CXXJUM^THRffi  COJUMNroUR 
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In  This  Group 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _  (Please  complete  the  table  below.) 

NO  _  (Please  turn  to  the  next  Task  Area.) 


OOUJMJOhC  CXXUMNIWO  COUUMN-TVfiffi  CXDUUMNRDUR 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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TASK  AREA  DATA 

Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 

YES _  (Please  complete  the  table  below.) 

NO  _ (Please  turn  to  the  next  Task  Area.) 


OOLUM^Oe 


How  Often 
Performs  Jobs 
In  This  Group 
(R/O/N) 


OOLUvmWO  CX3JLJMM'rH=1ffi 


Benchmark 
Worker 
(Check  One) 


Performance 

Time 


OOLJLJMNR3UR 


Supervision 

Time 


Hours  Minutes 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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TASK  AREA  DATA 


Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _ (Please  complete  the  table  below.) 

NO  _  (Please  turn  to  the  next  Task  Area.) 


OOLUM^C^E  OOUJM^TWD  OOLUM^TTRE  COUMNRDUR 
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How  Often 
Performs  Jobs 
In  This  Group 
(R/O/N) 
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This  Task  Group  includes  the  following  tasks  and  performance  benchmarks: 
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Isolate  Malfunctions  in  VHP  AM  Receiver-Transmitters  39  Mins. 

Isolate  Malfunctions  in  VHP  AM  Control  Units  18  Mins. 

Set-Up  VHP  AM  Peculiar  Test  Equipment  1 4  Mins. 


Are  any  of  the  tasks  on  the  previous  page  performed  by  any  of  the  workers  you 
supervise? 


YES _ (Please  complete  the  table  below.) 

NO  ___  (Please  turn  to  the  next  Task  Group.) 


OQLlJM^aE  OCLLM^TVC  OCULM^THS  OOHUtiKXB 
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How  Often 
Performs  Jobs 
In  This  Group 
(R/O/N) 
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WCfySH  IDHvmRCATiaN  INFDHMATTUN 


Worker 

How  long  in 
AFS? 

Social  Security  Number  (SSN) 

Name 

(Last,  First,  Ml) 

Worker 

Years 

Months 

1 
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APPENDIX  C:  TTP  DATA  COLLECTION  SITES 


Military  Airlift  Command 
Norton  AFB  CA 
McGuire  AFB  NJ 
Charleston  AFB  NC 
Travis  AFB  CA 
McChord  AFB  WA 
Altus  AFB  OK 

Strategic  Air  Command 
March  AFB  CA 
Minot  AFB  NO 
Beale  AFB  CA 
Loring  AFB  ME 
Blythesville  AFB  AK 
Castle  AFB  CA 

Tactical  Air  Command 
Homestead  AFB  FL 
Moody  AFB  GA 
Bergstrom  AFB  TX 
Seymour  Johnson  AFB  NC 
Shaw  AFB  NC 
George  AFB  CA 


